Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ The long-term goal: **generate artwork when necessary, animate and reuse it inde
- An optional **experimental** short-fade profile for imported half/closed eye drawings; avoids prolonged duplicate lash outlines at the cost of faster pose changes, revokes approval and regenerates review sheets.
- Local PNG pixel and ordered-frame QA (full frame or explicit region) reports RGB activity, unexpected transparency, candidate motion jumps and changed-control stalls; these metrics never auto-approve a rig.
- Import review can now render a 25-frame blink contact sheet focused on the actual eyelid transition boundaries, with pixel/motion diagnostics that never auto-approve a rig; see [Character Review](docs/CHARACTER_REVIEW.md).
- Imported flat-image actors start with narrower editable head/gaze motion limits, with consistent vertical look direction and unchanged amplitude for existing layered rigs; see [Look Motion](docs/LOOK_MOTION.md).
- Optional SHA-256 Rhubarb cue bindings can detect when audio or mouth-cue bytes are replaced after alignment; a bound scene fails validation/render instead of silently using stale lip sync.
- Agent-readable CLI output, tests and CI.

Expand Down
4 changes: 3 additions & 1 deletion docs/CHARACTER_REVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,4 +66,6 @@ Approval records a **user acknowledgement**, not proof of visual suitability. Ch

On 2026-11-07, the short-fade Hana trial retained valid source/rig fingerprint integrity but **remained unapproved**. A 25-frame blink review rendered successfully, found control-motion plateaus and showed unnatural skin-tone patches and doubled eyelid contours at transition frames. The short-fade profile reduced one type of overlap but did not repair the underlying drawings. The ability to detect and inspect these issues is real; attractive universal blinking from arbitrary flat illustrations is still an open milestone.

Future work: human-quality intermediate eyelid drawings, review-safe eye/head motion envelopes, additional licensed character styles and realistic mouth/voice assessment.
The latest imported-character build uses deliberately smaller head/eye look amplitudes; configure the specific character after inspecting the head and gaze sheets, as described in [LOOK_MOTION.md](LOOK_MOTION.md). Small limits do not repair torn eyelid sprites or certify visual suitability.

Future work: human-quality intermediate eyelid drawings, properly reviewed eye/head motion envelopes across styles, additional licensed character styles and realistic mouth/voice assessment.
1 change: 1 addition & 0 deletions docs/IMPLEMENTATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@
- [ ] Add head/neck seam-specific QA and review multiple rights-cleared character styles; metrics must never auto-approve a rig.
- [x] Wrap local Rev2D drawn eye/mouth variant request/status/import with explicit reviewed-art selection, approval revocation before import, post-import provenance refresh, and new review sheets. A real Hana avatar produced correct edit masks; unchanged fake drawings were rejected.
- [x] Test and expose an opt-in, validated short eye-sprite fade as a local derived-model experiment. Compare 13 actual Hana blink frames with native, immediate-switch and shortened fades: shorter fades visibly reduce double lashes at sampled midpoints, but introduce faster pose changes; never auto-approve.
- [x] Add bounded per-character semantic head/eye look amplitudes with conservative imported-PNG starting values, schema validation, compiler tests and real Hana ±yaw/pitch/gaze frame comparisons. This does not establish visual approval.
- [ ] Produce and visually approve high-quality closed/half eye and mouth artwork; demonstrate seamless real rendered transitions across full blink playback and multiple character styles. The local proof drawing stays unapproved.

## First video milestone — reproducible usable slice (2026-11-07)
Expand Down
37 changes: 37 additions & 0 deletions docs/LOOK_MOTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Authoring reviewable head and eye movement

PuppetControl's semantic scene `look` event uses normalized **screen-space** coordinates: `x=1` looks right, `x=-1` looks left, `y=1` looks **down**, and `y=-1` looks **up**. Both the head and irises follow the same direction. Rev2D itself declares positive `AngleY` and `EyeBallY` as up, so the compiler reverses both when scene `y` is positive.

## Per-character look limits

An imported single-image actor gets conservative **starting amplitudes** in its generated `character.json`:

~~~json
"lookLimits": {
"headYawDegrees": 8,
"headPitchDegrees": 5,
"eyeX": 0.2,
"eyeY": 0.2
}
~~~

The property lives inside the `render` object, alongside `expressionPresets`. These are **only intensity limits, not quality scores or proof of approval**. A character may still have broken eyelids, detached hair or an ugly neck at an 8-degree head turn. Always inspect actual pose sheets and video.

Before this feature, a `look` event at the extremes drove `AngleX=±22`, `AngleY=±12`, `EyeBallX=±0.5`, and `EyeBallY=±0.5`. Existing fully authored or upstream pre-rigged actors without `render.lookLimits` keep these original amplitudes. Previous provenance-tracked flat-image imports that lack the new field automatically use the conservative imported profile at render time. New `character build` manifests include the values explicitly.

All four limits are mandatory when the object is present. They are independently customizable in `character.json`, but can only **decrease** from the wider pre-rigged defaults. Setting any limit to `0` disables that channel for semantic `look` events. `character inspect <character.json>` shows the currently applied values and whether they came from the manifest, imported fallback, or legacy pre-rigged fallback.

For example, `look: { x: 0.5, y: -0.25 }` with the imported profile compiles to `AngleX=4`, `AngleY=+1.25`, `EyeBallX=+0.1`, `EyeBallY=+0.05`. The keyframes are eased over the event's specified duration, and expressions/mouth cues are unaffected.

## How to judge a setting

1. Render the imported character's head turn and gaze review sheets, and inspect small and extreme positions. Check for iris clipping, asymmetry, detached hair, shifting neck outlines and unintended background movement.
2. Start with the small imported amplitudes and only increase a channel after comparing actual frames of the same character. Reduce or disable channels that fail.
3. Use two `look` events in an editable scene, then render the whole transition at the target FPS. A clean endpoint does not prove intermediate frames are smooth.
4. **Keep the avatar unapproved** when any required pose is visually unacceptable. Altering the authored look configuration after approval warrants renewed review; the present asset hashes track PNG/landmarks/model/textures, not arbitrary manifest settings.

On 2026-11-07, the reviewed **experimental, unapproved** Hana import was directly rendered at yaw 22°/8°, pitch 12°/5° and gaze 0.5/0.2. The small states remain recognizable and exhibit less drastic movement, but the character still has unacceptable eyelid art and does not pass visual acceptance. The preview outputs are ignored under `outputs/qa/look-envelope-20261107`.

## Current boundary

This is a deliberate constraint on *how far* existing rig controls move. It does not create new eye sprites, compensate 3D head parallax, decompose layers or invent a physically believable pose transition. Per-character values are editable human/agent authored metadata; future work should associate a motion envelope's approval with its reviewed frame evidence.
4 changes: 3 additions & 1 deletion src/character/avatar.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { readFile, realpath, stat, lstat, mkdir, writeFile, rename, unlink } fro
import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
import { promisify } from "node:util";
import { characterSchema, sceneSchema, type Character, type Scene } from "../scene/schemas.js";
import { IMPORTED_INITIAL_LOOK_LIMITS } from "../scene/look-limits.js";
import { AVATAR_BUILD_RECORD, createAvatarBuildRecord, writeAvatarBuildRecord, revokeAvatarApprovalBeforeBuild } from "./provenance.js";
import { assessAvatarGeometry } from "./geometry-quality.js";

Expand Down Expand Up @@ -102,7 +103,8 @@ export function createImportedCharacter(id: string, model: Rev2DModelParameters)
expressions: Object.keys(expressionPresets),
motions: [],
transitions: [],
render: { backend: "rev2d", modelPath: id + ".r2d.json", expressionPresets },
render: { backend: "rev2d", modelPath: id + ".r2d.json", expressionPresets,
lookLimits: { ...IMPORTED_INITIAL_LOOK_LIMITS } },
};
return characterSchema.parse(manifest);
}
Expand Down
9 changes: 8 additions & 1 deletion src/cli/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import { requestDrawnVariants, statusDrawnVariants, importDrawnVariants } from "
import { comparePngFiles, parsePixelRegionArgument } from "../character/pixel-qa.js";
import { comparePngSequence } from "../character/frame-sequence-qa.js";
import { createBlinkReview } from "../character/blink-review.js";
import { DEFAULT_LOOK_LIMITS, IMPORTED_INITIAL_LOOK_LIMITS } from "../scene/look-limits.js";
import { createCueBinding, verifyCueBinding, CueBindingError } from "../audio/cue-provenance.js";

function print(value: unknown): void {
Expand Down Expand Up @@ -285,7 +286,13 @@ async function run(): Promise<void> {
expressions: parsed.data.expressions,
motions: parsed.data.motions,
transitions: parsed.data.transitions,
note: "Schema validation does not prove that a visual rig/asset can render.",
...(parsed.data.render ? {
lookLimits: parsed.data.render.lookLimits ?? (parsed.data.provenance
? IMPORTED_INITIAL_LOOK_LIMITS : DEFAULT_LOOK_LIMITS),
lookLimitsSource: parsed.data.render.lookLimits ? "manifest" : parsed.data.provenance
? "imported-initial-fallback" : "legacy-layered-fallback",
} : {}),
note: "Look limits bound movement, not visual quality; approval still requires real frame inspection.",
});
return;
}
Expand Down
25 changes: 19 additions & 6 deletions src/render/scene-plan.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import type { Character, Scene, SceneEvent } from "../scene/schemas.js";
import type { RhubarbMouthCue } from "../audio/rhubarb.js";
import { mapRhubarbCuesToRev2DVisemes } from "../audio/visemes.js";
import { DEFAULT_LOOK_LIMITS, IMPORTED_INITIAL_LOOK_LIMITS } from "../scene/look-limits.js";

export interface AnimationKey {
t: number;
Expand Down Expand Up @@ -30,8 +31,13 @@ export class PreviewUnsupportedError extends Error {
type ExpressionEvent = Extract<SceneEvent, { kind: "expression" }>;
type LookEvent = Extract<SceneEvent, { kind: "look" }>;

const HEAD_YAW_LIMIT = 22;
const HEAD_PITCH_LIMIT = 12;



/** Preserve exact zero for disabled look channels (never emit -0). */
function scaledLook(value: number, limit: number): number {
return value === 0 || limit === 0 ? 0 : value * limit;
}

/** Piecewise linear evaluation is only used when selecting a new animation endpoint. */
function valueAt(keys: AnimationKey[], timestamp: number): number {
Expand Down Expand Up @@ -209,11 +215,18 @@ export function compilePerformance(
if (event.kind === "look") {
const start = event.atMs / 1000;
const end = Math.min(durationSeconds, (event.atMs + event.durationMs) / 1000);
// Imported flat art starts with restrained motion; reviewers can edit
// this envelope after inspecting actual eye, hair and neck poses.
// Pre-rigged layered characters keep the wider legacy mapping.
const bounds = render.lookLimits ?? (character.provenance
? IMPORTED_INITIAL_LOOK_LIMITS : DEFAULT_LOOK_LIMITS);
const mappings = {
AngleX: event.x * HEAD_YAW_LIMIT,
AngleY: -event.y * HEAD_PITCH_LIMIT,
EyeBallX: event.x * 0.5,
EyeBallY: event.y * 0.5,
AngleX: scaledLook(event.x, bounds.headYawDegrees),
AngleY: scaledLook(-event.y, bounds.headPitchDegrees),
EyeBallX: scaledLook(event.x, bounds.eyeX),
// Scene coordinates are screen-space (+y down); Rev2D +Y is up.
// Head and eyes must agree vertically instead of counter-steering.
EyeBallY: scaledLook(-event.y, bounds.eyeY),
};
for (const [parameter, goal] of Object.entries(mappings)) {
if (!rigDefaults.has(parameter)) {
Expand Down
34 changes: 34 additions & 0 deletions src/scene/look-limits.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
/**
* Semantic look controls -> Rev2D rig parameters.
*
* These are *amplitude limits*, not an assessment that an illustration looks
* good at any particular pose. Every imported flat PNG needs actual visual
* review of iris clipping and head/neck distortion.
*/
export interface LookLimits {
headYawDegrees: number;
headPitchDegrees: number;
eyeX: number;
eyeY: number;
}

export const DEFAULT_LOOK_LIMITS: Readonly<LookLimits> = Object.freeze({
headYawDegrees: 22,
headPitchDegrees: 12,
eyeX: 0.5,
eyeY: 0.5,
});

/**
* Conservative initial envelope for imported single-layer illustrations.
*
* Not artist-approved. This is deliberately smaller than the engine's
* supported parameter bounds and should be changed only after reviewing
* actual parameter sweeps for the specific character.
*/
export const IMPORTED_INITIAL_LOOK_LIMITS: Readonly<LookLimits> = Object.freeze({
headYawDegrees: 8,
headPitchDegrees: 5,
eyeX: 0.2,
eyeY: 0.2,
});
9 changes: 9 additions & 0 deletions src/scene/schemas.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { z } from "zod";
import { DEFAULT_LOOK_LIMITS } from "./look-limits.js";

const identifier = z.string().trim().min(1).max(100).regex(/^[a-zA-Z0-9][a-zA-Z0-9_-]*$/);
const milliseconds = z.number().int().nonnegative();
Expand Down Expand Up @@ -38,6 +39,13 @@ export const characterSchema = z.strictObject({
expressionPresets: z.record(z.string(), z.record(z.string(), z.number())),
/** Controls for multiple poses represented by one deformable rig. */
posePresets: z.record(z.string(), z.record(z.string(), z.number())).optional(),
/** Authored look amplitudes; limits alone do not certify visual quality. */
lookLimits: z.strictObject({
headYawDegrees: z.number().finite().min(0).max(DEFAULT_LOOK_LIMITS.headYawDegrees),
headPitchDegrees: z.number().finite().min(0).max(DEFAULT_LOOK_LIMITS.headPitchDegrees),
eyeX: z.number().finite().min(0).max(DEFAULT_LOOK_LIMITS.eyeX),
eyeY: z.number().finite().min(0).max(DEFAULT_LOOK_LIMITS.eyeY),
}).optional(),
}).optional(),
id: identifier,
defaultPose: identifier,
Expand Down Expand Up @@ -98,6 +106,7 @@ export const sceneEventSchema = z.discriminatedUnion("kind", [
z.strictObject({
...actorEvent,
kind: z.literal("look"),
/** Normalized screen-space target: +x right, +y down. */
x: z.number().min(-1).max(1),
y: z.number().min(-1).max(1),
durationMs: milliseconds.default(250),
Expand Down
3 changes: 3 additions & 0 deletions tests/avatar-import.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ describe("agent-first avatar import contracts", () => {
expect(result.expressions).toEqual(["neutral", "happy", "confused"]);
expect(result.render?.expressionPresets.happy).toEqual({ MouthForm: 0.6 });
expect(result.render?.expressionPresets.confused).toEqual({ MouthForm: -0.3 });
expect(result.render?.lookLimits).toEqual({
headYawDegrees: 8, headPitchDegrees: 5, eyeX: 0.2, eyeY: 0.2,
});
expect(result.transitions).toEqual([]);
});

Expand Down
61 changes: 61 additions & 0 deletions tests/scene-plan.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,67 @@ describe("performance compiler (one rig)", () => {
]);
});

it("keeps legacy look amplitude for existing layered characters", () => {
const animation = compilePerformance(build([
{ atMs: 200, actor: "hana", kind: "look", x: 1, y: -1, durationMs: 400 },
]), actor, "hana", defaultRig);
const values = new Map(animation.tracks.map((track) =>
[track.target, track.keys.find((key) => key.t === 0.6)?.v]));
expect(values.get("param:AngleX")).toBe(22);
expect(values.get("param:AngleY")).toBe(12);
expect(values.get("param:EyeBallX")).toBe(0.5);
expect(values.get("param:EyeBallY")).toBe(0.5);
});

it("restrains look for legacy flat-image imports even without a new manifest field", () => {
const imported = characterSchema.parse({
...actor, provenance: { file: "avatar-build.json" },
});
const animation = compilePerformance(build([
{ atMs: 200, actor: "hana", kind: "look", x: 1, y: -1, durationMs: 400 },
]), imported, "hana", defaultRig);
const values = new Map(animation.tracks.map((track) =>
[track.target, track.keys.find((key) => key.t === 0.6)?.v]));
expect(values.get("param:AngleX")).toBe(8);
expect(values.get("param:AngleY")).toBe(5);
expect(values.get("param:EyeBallX")).toBe(0.2);
expect(values.get("param:EyeBallY")).toBe(0.2);
});

it("obeys each explicitly authored look limit independently, including zero", () => {
const authored = characterSchema.parse({
...actor, render: {
...actor.render,
lookLimits: { headYawDegrees: 4, headPitchDegrees: 1,
eyeX: 0, eyeY: 0.1 },
},
});
const animation = compilePerformance(build([
{ atMs: 200, actor: "hana", kind: "look", x: -1, y: 1, durationMs: 400 },
]), authored, "hana", defaultRig);
const values = new Map(animation.tracks.map((track) =>
[track.target, track.keys.find((key) => key.t === 0.6)?.v]));
expect(values.get("param:AngleX")).toBe(-4);
expect(values.get("param:AngleY")).toBe(-1);
expect(values.get("param:EyeBallX")).toBe(0);
expect(values.get("param:EyeBallY")).toBe(-0.1);
});

it("rejects malformed or wider-than-supported look limits", () => {
const limits = { headYawDegrees: 8, headPitchDegrees: 5, eyeX: 0.2, eyeY: 0.2 };
for (const bad of [
{ ...limits, eyeX: 0.8 },
{ ...limits, headYawDegrees: -1 },
{ ...limits, headPitchDegrees: 30 },
{ ...limits, eyeY: Number.POSITIVE_INFINITY },
{ eyeX: 0.2 },
{ ...limits, extra: true },
]) {
expect(characterSchema.safeParse({ ...actor, render: { ...actor.render, lookLimits: bad } }).success)
.toBe(false);
}
});

it("rejects unsupported pose changes rather than faking a seamless transition", () => {
expect(() => compilePerformance(build([
{ atMs: 300, actor: "hana", kind: "pose", name: "crossed" },
Expand Down
Loading