diff --git a/.agents/skills/puppetcontrol/SKILL.md b/.agents/skills/puppetcontrol/SKILL.md index 7cba3d2..2f72b8e 100644 --- a/.agents/skills/puppetcontrol/SKILL.md +++ b/.agents/skills/puppetcontrol/SKILL.md @@ -13,11 +13,14 @@ Read repo-root AGENTS.md, docs/TECHNICAL_CONCEPT.md, and docs/CHARACTER_IMPORT.m 4. For optional image-model-generated landmarks, run `node dist/cli/index.js character landmarks propose --provider codex` only when the user accepts sending artwork to their configured Codex provider. The command produces a review candidate or diagnostic output, never an approved rig. Otherwise, for a manual agent task, run `node dist/cli/index.js character landmarks assist `. Open source.png, work/grid.png and work/landmark-agent-task.md. Fill the generated JSON proposal template with original source-pixel coordinates; do not guess from resized images. Read .cache/rev2d/docs/AVATAR.md. 5. Stage the proposal using `node dist/cli/index.js character landmarks stage --proposal `. Rev2D validates the isolated candidate and returns a work/overlay.png. **Actually inspect the overlay** before `character landmarks accept --candidate --reviewed`. Accepting replaces avatar.json with a backup and revokes all existing pose approvals. Never auto-pass --reviewed solely because a validator passed. 6. Build using node dist/cli/index.js character build (use --force if prior output exists). Inspect any geometryWarnings, particularly BODY_PIVOT_TOO_LOW and IRIS_LANDMARK_MISSING; these do not replace pixel inspection. Source/landmark/model/texture hashes are recorded in avatar-build.json and scene preflight rejects stale assets. Inspect every returned review/poses.png, turn.png, eyes.png and mouth.png. If incorrect, repair avatar.json and rebuild using --force; this resets the generated manifest and any edited scaffold. -7. A successful build intentionally leaves character.json with approved:false. A --force rebuild revokes approval *before* writing model/texture files, even if the rebuild later fails. Only after actual visual QA may an agent/user explicitly run `node dist/cli/index.js character approve --reviewed` (the flag is an acknowledgement, not evidence of review) to record neutral-pose approval. If there are torn eyelids, missing areas, extra contours or identity drift, LEAVE IT UNAPPROVED and report what needs correction. -8. For a reviewed reusable rig, use `node dist/cli/index.js scene new --character --seconds 12 --fps 15` to create a strict editable scene. Reuse the same character file in a second scene without rebuilding artwork. The generated preview-scene.json is also editable. Use node dist/cli/index.js scene validate and scene plan before rendering. Preview PNG frames, inspect key timestamps, then render and inspect the MP4. -9. Within a compatible shared rig, semantic pose events may interpolate through validated parameter presets (the neutral-to-pointing demo uses shoulder and elbow controls). Distinct pose models require explicit scene camera cuts; an authored bridge is NOT implemented. Never silently crossfade meshes or call a body cut a smooth physical transition. -10. For speech, prepare WAV, install Rhubarb separately, and use node dist/cli/index.js audio align --out [--binary ]. A `scene new` voice track requires both `--voice` and the actual `--cues` file; no fabricated speech sync. Reference audio.path, audio.startMs and audio.cuesPath from the scene; review the actual mouth/audio timing. -11. Do not commit private character illustrations, full generated models, user voice recordings or secrets without checking redistribution rights. Demo:avatar generates pinned MIT-licensed sample artwork into ignored .cache. -12. State what actually rendered and which controls remain unsupported. Do not claim a flat PNG is automatically converted into production-quality animation without landmark review. +7. To test optional drawn eye/mouth variants on a built actor, request edit masks, locally supply reviewed `variants/.png`, and explicitly import. With reviewed closed+half eye drawings, `character variants import eyes_closed eyes_half --experimental-short-fade --reviewed-art` shortens fade windows but makes poses change faster and may still ghost; it is a visual experiment, never a default or automatic quality fix. The import revokes prior approval and refreshes provenance only after re-rendering review sheets. +8. Before approving a verified imported rig, run `node dist/cli/index.js character qa blink ` to generate an ordered 25-frame actual eyelid transition contact sheet (`blink-eyes.png` and `blink-full.png`); inspect EyeOpen=0.19 and 0.69 and the real eye artwork seams. The QA report flags motion stalls but never passes visual approval. +9. A successful build intentionally leaves character.json with approved:false. A --force rebuild revokes approval *before* writing model/texture files, even if the rebuild later fails. Only after actual visual QA may an agent/user explicitly run `node dist/cli/index.js character approve --reviewed` (the flag is an acknowledgement, not evidence of review) to record neutral-pose approval. If there are torn eyelids, missing areas, extra contours or identity drift, LEAVE IT UNAPPROVED and report what needs correction. +10. For a reviewed reusable rig, use `node dist/cli/index.js scene new --character --seconds 12 --fps 15` to create a strict editable scene. Reuse the same character file in a second scene without rebuilding artwork. The generated preview-scene.json is also editable. Use node dist/cli/index.js scene validate and scene plan before rendering. Preview PNG frames, inspect key timestamps, then render and inspect the MP4. +11. Within a compatible shared rig, semantic pose events may interpolate through validated parameter presets (the neutral-to-pointing demo uses shoulder and elbow controls). Distinct pose models require explicit scene camera cuts; an authored bridge is NOT implemented. Never silently crossfade meshes or call a body cut a smooth physical transition. +12. For speech, prepare WAV, install Rhubarb separately, and use node dist/cli/index.js audio align --out [--binary ]. For exact-byte continuity, optionally run `audio bind --cues --out ` and include `--binding ` in `scene new`; bound scenes fail validation/render if cue or voice bytes change. A `scene new` voice track requires both `--voice` and the actual `--cues` file; no fabricated speech sync. Reference audio.path, audio.startMs and audio.cuesPath from the scene; review the actual mouth/audio timing. +13. Do not commit private character illustrations, full generated models, user voice recordings or secrets without checking redistribution rights. Demo:avatar generates pinned MIT-licensed sample artwork into ignored .cache. +14. For animation QA, compare real ordered PNGs with `node dist/cli/index.js character qa sequence ... [--region x,y,w,h] [--control-values v0,v1,...] [--cut-after i,j]`. Supply sampled control values when investigating invisible or stalled motion; label known cuts. Inspect the PNGs/video and report possible ghosting even if the motion statistics look normal. The QA report is not an approval. +15. State what actually rendered and which controls remain unsupported. Do not claim a flat PNG is automatically converted into production-quality animation without landmark review. Keep edits small and testable. Run npm run verify and inspect visual evidence for any animation/asset changes. \ No newline at end of file diff --git a/README.md b/README.md index 3fbdc10..1efcd42 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,11 @@ The long-term goal: **generate artwork when necessary, animate and reuse it inde - An opt-in `character approve --reviewed` gate for visually inspected imported rigs; asset fingerprints and review sheets are checked before approval. - `scene new` creates editable expression/look/dialogue scenes from an already approved rig without changing the character; voice requires genuine Rhubarb cues. - Non-destructive MP4 export: output files are promoted only after successful FFmpeg encoding, with overwrite requiring explicit `--force`. +- Guarded **local drawn eye/mouth variant** request, status and import commands using Rev2D masks and sprites. Imported drawings revoke approval and require a fresh visual review; no image generation happens automatically. +- An optional **experimental** short-fade profile for imported half/closed eye drawings; avoids prolonged duplicate lash outlines at the cost of faster pose changes, revokes approval and regenerates review sheets. +- Local PNG pixel and ordered-frame QA (full frame or explicit region) reports RGB activity, unexpected transparency, candidate motion jumps and changed-control stalls; these metrics never auto-approve a rig. +- Import review can now render a 25-frame blink contact sheet focused on the actual eyelid transition boundaries, with pixel/motion diagnostics that never auto-approve a rig; see [Character Review](docs/CHARACTER_REVIEW.md). +- Optional SHA-256 Rhubarb cue bindings can detect when audio or mouth-cue bytes are replaced after alignment; a bound scene fails validation/render instead of silently using stale lip sync. - Agent-readable CLI output, tests and CI. **Not implemented yet:** fully automatic landmark placement or reliable one-click rigging, AI-generated arbitrary new pose rigs, convincing smooth bridges between unrelated meshes (especially crossed arms), automatic transcription/TTS, captions, B-roll and multiple actors. The arm-raise gesture is genuinely animated, but its artwork is still prototype-quality. Other pose families may require edited hard cuts. The pose manifest is capability metadata; visual poses become usable only after their rig files are generated, validated and inspected. @@ -48,6 +53,10 @@ npm run demo:avatar # Import a freshly rendered flat PNG and create an unappr npm run demo:first-video # Prepare an ignored, unapproved Hana fixture for review # After actually reviewing the pinned Hana test fixture only: # node scripts/demo-first-video.mjs --reviewed-sample +# For a built imported avatar, see docs/CHARACTER_IMPORT.md for drawn-variant request/import +# Compare captured PNG frames: +# node dist/cli/index.js character qa compare outputs/rest.png outputs/closed.png --region 175,220,170,75 +# node dist/cli/index.js character qa sequence outputs/f00.png outputs/f01.png outputs/f02.png --control-values 1,0.5,0 ~~~ The sample uses Rev2D's own **Hana** rig from the pinned MIT-licensed upstream checkout; generated character rig JSON is not checked into this repository. The derived example video is separately attributed and licensed under the original Rev2D terms. @@ -102,6 +111,7 @@ Our native animation backend for the spike is **Rev2D**, pinned by the setup hel - [Agent-guided landmark review and acceptance](docs/LANDMARK_AGENT_WORKFLOW.md) - [Optional Codex vision inference: real-image results and limitations](docs/CODEX_VISION_LANDMARKS.md) - [Deep quality and reliability audit with observed pixel measurements](docs/QUALITY_AUDIT.md) +- [Pixel-level visual QA (non-approval diagnostics)](docs/PIXEL_QA.md) - [Architecture decision log](docs/DECISIONS.md) Audio/lip-sync works with precomputed Rhubarb cues and a locally installed executable. The smooth pointing demo reuses the same face and deforms one rig continuously; crossed arms still need a more convincing bridge or intentional cut. Next: polish eyelid and mouth deformation by reusing Rev2D's reviewed drawn variants, expand automated image-quality probes and evaluate diverse source artwork. Do not confuse data validation with visual quality. diff --git a/docs/CHARACTER_IMPORT.md b/docs/CHARACTER_IMPORT.md index 35913c5..c89857c 100644 --- a/docs/CHARACTER_IMPORT.md +++ b/docs/CHARACTER_IMPORT.md @@ -77,6 +77,38 @@ node dist/cli/index.js scene render avatars/portrait/preview-scene.json --out ou Scene validation also checks avatar-build.json against source.png, avatar.json, the built model and all generated textures. Changing any of them requires a new build and renewed review; stale files fail with AVATAR_ASSETS_STALE. Before approval, the scene intentionally fails with DEFAULT_POSE_NOT_APPROVED. The built model begins with one body pose family (neutral), not automatic pointing or crossed-arms artwork. Its provenance record covers that primary rig only: adding a different pose model to this imported actor is rejected until separate pose-model fingerprints are supported. Manually authored multi-pose characters remain available through the existing explicit rig workflow. +## Improve eyes and mouth with drawn variants (experimental) + +If the automatically deformed eyelids or mouth look smeared, use **Rev2D's existing drawn sprites**. This does not automatically create beautiful art: someone must supply a carefully edited copy of the original PNG. The renderer falls back to mesh deformation for shapes without an accepted drawing. + +~~~sh +node dist/cli/index.js character variants request avatars/portrait eyes_closed eyes_half mouth_a +node dist/cli/index.js character variants status avatars/portrait +~~~ + +The request produces `variant-requests/.mask.png` and `variant-requests/.prompt.txt` beside the original `source.png`. The mask's opaque black pixels must remain identical; only its editable, transparent area should be redrawn. Save each **full-size** corrected illustration as `variants/.png` (for example `variants/eyes_closed.png`). A person or an image generator with **explicit artwork/privacy permission** may produce these PNGs; **PuppetControl's variant commands make no external image requests**. + +After examining each candidate at full resolution, import only those exact IDs: + +~~~sh +node dist/cli/index.js character variants import avatars/portrait eyes_closed eyes_half mouth_a --reviewed-art +node dist/cli/index.js character variants status avatars/portrait +~~~ + +For illustrations with **both** `eyes_closed` and `eyes_half` correctly drawn, there is an additional **opt-in experiment**: + +~~~sh +node dist/cli/index.js character variants import avatars/portrait eyes_closed eyes_half --experimental-short-fade --reviewed-art +~~~ + +The default import preserves Rev2D's native fades. The experimental switch shortens the eye-sprite crossfade windows to `EyeOpen 0.18–0.20` (closed to half) and `0.68–0.70` (half to mesh). This reduces double outlines in some frames, **but makes the eyelids pop more quickly**, and ghosting remains possible within the short transition windows. It requires both drawn eye shapes on both sides and refuses unsupported/smiling-eye setups. A later import **without** the flag returns to the native generated fades. This is not a production-quality guarantee. + +Import checks the drawing size, then delegates to Rev2D's mask, global shift/zoom and seam checks. It revokes every existing pose approval **before** touching model or texture assets; on success it re-renders `review/eyes.png`, `review/mouth.png` and `review/poses.png` and refreshes asset fingerprints. **It never auto-approves.** Inspect the output for face jumps, halos, duplicate lines and inconsistent colors, then explicitly reapprove in `character.json` if acceptable. + +An upstream import can partially accept images before rejecting another. On error, the character remains unapproved and the old provenance may intentionally be stale; rebuild with `character build avatars/portrait --force` if necessary, inspect again, and only then retry. Do not assume that a successful import means the new drawings look good. Automatic image generation, reliable facial polish and smooth unrelated-pose bridges remain separate goals. + +For a detailed eyelid/mouth correction procedure and a 25-frame blink contact sheet, see [CHARACTER_REVIEW.md](CHARACTER_REVIEW.md). Use `node dist/cli/index.js character qa blink avatars/portrait` before approval; it never modifies the rig or approves it. The original imported Hana trial still exhibits significant seams and is not acceptable character art. + ## Reproducible built-in demonstration ~~~sh diff --git a/docs/CHARACTER_REVIEW.md b/docs/CHARACTER_REVIEW.md new file mode 100644 index 0000000..1f418d3 --- /dev/null +++ b/docs/CHARACTER_REVIEW.md @@ -0,0 +1,69 @@ +# Character review: from flat PNG to a reusable actor + +This is a **guided visual-correction workflow**, not a one-click promise. PuppetControl can import a front-facing PNG and prepare a Rev2D rig, but source landmarks and sprite artwork may need manual or agent-assisted repairs. **Never approve a rig merely because a CLI validation or pixel report passes.** + +## 1. Import, annotate and build + +Run commands inside the PuppetControl workspace: + +~~~powershell +node dist/cli/index.js character import art/portrait.png --out avatars/portrait --id portrait +node dist/cli/index.js character landmarks assist avatars/portrait +node dist/cli/index.js character build avatars/portrait +~~~ + +Inspect `avatars/portrait/work/grid.png` and the accepted `avatar.json` landmark overlay. Correct any erroneous eyes, neck, head or body pivot before building. Use the existing staged landmark-proposal workflow for agent-proposed points. Imported actors remain `approved:false`. + +## 2. Request optional real eyelid and mouth drawings + +When flat-image eyelid deformation is visibly wrong: + +~~~powershell +node dist/cli/index.js character variants request avatars/portrait eyes_closed eyes_half mouth_a +node dist/cli/index.js character variants status avatars/portrait +~~~ + +The requests produce PNG masks and text instructions under `variant-requests/`. Provide **full-canvas PNGs** at `variants/eyes_closed.png`, `variants/eyes_half.png`, etc. Modify only the mask area, match the original skin/outline pixels at the seam, and keep the rest of the character byte-for-byte visually consistent. The validator rejects many unchanged or misaligned edits but cannot tell if eyelashes look natural. + +Only after locally reviewing the submitted artwork, import the selected IDs: + +~~~powershell +node dist/cli/index.js character variants import avatars/portrait eyes_closed eyes_half --reviewed-art +~~~ + +An **optional** short-fade experiment can reduce double contours by narrowing sprite opacity blends. It is not always smoother and may make the eyelids pop: + +~~~powershell +node dist/cli/index.js character variants import avatars/portrait eyes_closed eyes_half --experimental-short-fade --reviewed-art +~~~ + +Both paths revoke any earlier approval and generate new review sheets and fingerprints. + +## 3. Inspect exact blink transition frames + +~~~powershell +node dist/cli/index.js character qa blink avatars/portrait +~~~ + +This produces an **ignored, uniquely named** directory under `outputs/qa/blink-review/` with `blink-full.png`, `blink-eyes.png`, `blink-review.json` and the individual PNG frames. The 25-frame sequence samples closed/open and both short-fade transition boundaries, including `EyeOpen=0.19` and `EyeOpen=0.69`. Thirteen actual Rev2D states are rendered; the remaining twelve are the same states in reverse order. Review the montage left-to-right, top-to-bottom. + +The JSON report lists candidate motion stalls, frame changes, sampled controls and crop coordinates. **None of those statistics establish good visual quality.** Scrutinize both eyelids for lingering lashes, matte seams, iris clipping, mismatched shadow/color and sudden double contours. Inspect the existing `review/eyes.png`, `review/mouth.png`, `review/turn.png` and `review/poses.png` too. If eye annotations are unavailable, specify `--region x,y,width,height` explicitly. + +## 4. Approve only after actual visual review + +Only when all relevant states look acceptable, run: + +~~~powershell +node dist/cli/index.js character approve avatars/portrait --reviewed +node dist/cli/index.js scene new .cache/portrait-first.json --character avatars/portrait/character.json --seconds 12 --fps 15 +node dist/cli/index.js scene validate .cache/portrait-first.json +node dist/cli/index.js scene render .cache/portrait-first.json --out outputs/portrait-first.mp4 +~~~ + +Approval records a **user acknowledgement**, not proof of visual suitability. Changing source art, landmarks, rig or textures invalidates the verified build record. Any new variant import resets approval. + +## What the actual Hana trial proves + +On 2026-11-07, the short-fade Hana trial retained valid source/rig fingerprint integrity but **remained unapproved**. A 25-frame blink review rendered successfully, found control-motion plateaus and showed unnatural skin-tone patches and doubled eyelid contours at transition frames. The short-fade profile reduced one type of overlap but did not repair the underlying drawings. The ability to detect and inspect these issues is real; attractive universal blinking from arbitrary flat illustrations is still an open milestone. + +Future work: human-quality intermediate eyelid drawings, review-safe eye/head motion envelopes, additional licensed character styles and realistic mouth/voice assessment. diff --git a/docs/FIRST_VIDEO.md b/docs/FIRST_VIDEO.md index eb651ac..9d4fcff 100644 --- a/docs/FIRST_VIDEO.md +++ b/docs/FIRST_VIDEO.md @@ -73,6 +73,13 @@ Then create the scene with both the WAV and cue JSON: node dist/cli/index.js scene new .cache/my-video/voiced.json --character path/to/character.json --id voiced --seconds 5 --fps 15 --size 512x640 --voice path/to/voice.wav --cues .cache/my-video/mouth.json +For extra protection against outdated lip sync, optionally bind exact voice and Rhubarb cue bytes before scene creation: + + node dist/cli/index.js audio bind path/to/voice.wav --cues .cache/my-video/mouth.json --out .cache/my-video/mouth.binding.json + node dist/cli/index.js audio verify path/to/voice.wav --cues .cache/my-video/mouth.json --binding .cache/my-video/mouth.binding.json + +Then pass `--binding .cache/my-video/mouth.binding.json` alongside `--voice` and `--cues` in `scene new`. The scene will fail validation and rendering if either file is changed or moved. Sidecar creation deliberately refuses overwriting an existing binding; use a new output name for a fresh alignment. Older unbound scenes remain supported, but they do not have this exact-byte protection. + The current first-video path requires --voice and --cues together. Validate and render the resulting scene normally. A real 12-second test using local nonredistributed voice input successfully generated 36 Rhubarb cues and a 180-frame H.264/AAC MP4 at 512×640, 15 FPS. The audio recording, cues and MP4 are intentionally kept only in ignored .cache/first-video-demo/qa-12s-voiced/. This proves file/cue/mux integration, not perceptual lip-sync quality on every speaker. ## Non-destructive rendering diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/IMPLEMENTATION_PLAN.md index 65ac334..24b77fa 100644 --- a/docs/IMPLEMENTATION_PLAN.md +++ b/docs/IMPLEMENTATION_PLAN.md @@ -76,8 +76,13 @@ - [x] Add conservative geometry triage for low torso pivot/fade, missing/outside-eye iris and shearing head pivot; return nonblocking warnings. - [x] Revoke existing approval before any --force build mutates Rev2D outputs, and fail closed on stale source/spec/model/texture fingerprints. - [x] Eliminate reproduced black edge artifacts caused by alpha-zero warped pixels in H.264: composite frames onto scene background, with a 60-frame before/after comparison. -- [ ] Add pixel-level motion QA probes and a reviewable report; automated geometry warnings are not proof of visual quality. -- [ ] Wrap Rev2D's existing drawn closed-eye/mouth variants to reduce visible deformation artifacts without writing another mesh engine. +- [x] Implement bounded PNG rest/activity region QA; metrics never prove visual quality. +- [x] Add 2–60 ordered-frame motion QA with optional ROI, control samples and intentional cut indices; verified on a 13-frame Hana blink cycle. +- [x] Add a reviewable 25-frame blink contact sheet and bounded motion report covering EyeOpen 0.18/0.19/0.20 and 0.68/0.69/0.70; actual imported Hana test remained unapproved. +- [ ] Add head/neck seam-specific QA and review multiple rights-cleared character styles; metrics must never auto-approve a rig. +- [x] Wrap local Rev2D drawn eye/mouth variant request/status/import with explicit reviewed-art selection, approval revocation before import, post-import provenance refresh, and new review sheets. A real Hana avatar produced correct edit masks; unchanged fake drawings were rejected. +- [x] Test and expose an opt-in, validated short eye-sprite fade as a local derived-model experiment. Compare 13 actual Hana blink frames with native, immediate-switch and shortened fades: shorter fades visibly reduce double lashes at sampled midpoints, but introduce faster pose changes; never auto-approve. +- [ ] Produce and visually approve high-quality closed/half eye and mouth artwork; demonstrate seamless real rendered transitions across full blink playback and multiple character styles. The local proof drawing stays unapproved. ## First video milestone — reproducible usable slice (2026-11-07) @@ -91,7 +96,8 @@ This milestone is deliberately **narrower** than M5's universal single-image pro - [x] Prevent silent overwrite of existing MP4s by encoding to a temporary sibling file; `--force` explicitly replaces only after successful encoding. - [x] Make `scene validate` distinguish planning-valid bridge/caption/motion documents from scenes the current renderer can actually export, with actionable errors. - [ ] Achieve visibly polished eye/mouth deformation and user-friendly assisted rig correction on **diverse** artwork; the imported Hana avatar previously showed artifacts and remains unapproved. -- [ ] Verify acceptable lip synchronization with multiple natural speakers, and bind Rhubarb cue data to its source-audio fingerprint. +- [x] Add optional exact-byte SHA-256 Rhubarb cue binding sidecars; verified bound first-video scene and live H.264/AAC render. Sidecars are opt-in, and legacy scenes remain supported. +- [ ] Verify visually acceptable lip synchronization with multiple natural speakers; optional integrity binding alone does not prove phoneme alignment. - [ ] Produce a convincing, reproducible 10–15s video **using an independently imported and genuinely approved custom illustration** without developer intervention. The 12s test so far uses a pre-rigged licensed sample. ## Acceptance benchmark diff --git a/docs/PIXEL_QA.md b/docs/PIXEL_QA.md new file mode 100644 index 0000000..db30b71 --- /dev/null +++ b/docs/PIXEL_QA.md @@ -0,0 +1,75 @@ +# Pixel-level visual QA (triage only) + +**Status:** local PNG comparison implemented with deterministic RGBA decoding through the already-required FFmpeg executable. This is **not** a visual-approval model and not a perceptual similarity metric. + +## CLI + +Capture two frame images of exactly the same canvas and resolution. For a Rev2D avatar, use the engine's `render` command (review the current pinned backend help for flags) or the generated scene preview PNGs. + +~~~sh +node dist/cli/index.js character qa compare outputs/rest.png outputs/eyes-closed.png +node dist/cli/index.js character qa compare outputs/rest.png outputs/eyes-closed.png --region 175,220,170,75 +~~~ + +The optional region is an **integer** `x,y,width,height` rectangle within both images. Always use comparable camera, framing and character pose settings; do not compare resized images or a camera cut to an unchanged rest pose and interpret the result as articulation. + +The output includes: + +| Field | Meaning | +| --- | --- | +| `rgbMeanAbsoluteError` | Average absolute difference per RGB channel, in 0–255 intensity levels | +| `changedPixels` and `changedFraction` | Pixels whose RGB or alpha changed by more than 10 levels; fraction of comparable ROI pixels | +| `ignoredTransparentPixels` | ROI pixels where both samples have negligible alpha and RGB is visually undefined | +| `unexpectedTransparentBorderPixels` | Full-image perimeter pixels that were opaque in the reference and became nonopaque in the candidate | +| `region` | Coordinates actually evaluated for motion and RGB differences | + +Inputs must be PNGs under 30 MB with dimensions 256–4096 pixels per side. Files must be inside the active workspace. PNGs with mismatched dimensions or invalid ROI are rejected rather than silently scaled. FFmpeg is invoked with fixed argument arrays, without a shell, and decoded RGBA buffers are bounded. + +## Real-data probe (Hana, 2026-11-07) + +A fresh local 512×640 Hana import was rendered at rest and again with both eye-open parameters set to 0, using the same native camera and an opaque background. The resulting source-versus-rest full-frame comparison gave `rgbMeanAbsoluteError = 0.0002`, `changedPixels = 0`, and `unexpectedTransparentBorderPixels = 0` for that specific imported test drawing. + +For a 170×75 eye-region crop, rest versus closed-eyes produced `rgbMeanAbsoluteError = 33.5879`, `changedFraction = 0.364392`, and `unexpectedTransparentBorderPixels = 0`. These numbers establish **pixel activity**, not the quality of the drawing. Human visual inspection of the same review sheets still showed flat half-eye artwork and double contour artifacts in fades. + +## Ordered-frame sequence QA (temporal motion) + +Provide **2–60 PNGs in time order**. This does not infer sorting from filenames. The CLI analyzes adjacent frames, excludes declared intentional cuts from its local comparison baseline and reports unusually large changes as candidates for visual inspection. + +~~~sh +node dist/cli/index.js character qa sequence outputs/f00.png outputs/f01.png outputs/f02.png outputs/f03.png --region 175,220,170,75 +node dist/cli/index.js character qa sequence outputs/f00.png outputs/f01.png outputs/f02.png outputs/f03.png --region 175,220,170,75 --control-values 1,0.7,0.3,0 +~~~ + +The optional `--control-values` list needs exactly one finite numeric control sample **per frame** and must describe a specific animated rig parameter (for example, `EyeLOpen`). When the control moves but rendered pixels barely change, the result is a `possibleMotionStalls` candidate. A static region, a clamped value, occlusion or an intentionally paused animation can also cause a flagged pair, so **visual review remains mandatory**. + +For edited scenes, supply zero-based `--cut-after` indices to identify intended jumps: for example, `--cut-after 2` means the change between input frames 2 and 3 is an editorial cut and should not be treated as a surprise jump. The CLI reports the per-pair differences and candidate indices; it **never** returns a binary animation-quality verdict. The `firstToLast` measurement helps verify loop returns when the endpoints should match, but a deliberately non-looping scene need not return to start. + +### Real 13-frame blink diagnosis (Hana, 2026-11-07) + +The previously generated, intentionally unapproved Hana sprite trial was rendered locally at 512×640 with `EyeLOpen` and `EyeROpen` stepped together from 1 to 0 and back to 1 in 13 frames. A silent 30fps MP4 and manually assembled contact sheet were inspected under ignored `outputs/qa/temporal-blink-20261107/`. The new report showed: + +| Metric | Observed | +| --- | --- | +| Adjacent pairs / frames | 12 / 13 | +| ROI | x 175, y 220, width 170, height 75 | +| Median adjacent RGB difference | 10.7044 / 255 | +| Potential abrupt pairs | none under generic outlier heuristic | +| Expected-control motion stalls | pairs 3→4 and 8→9 (0.166667 control delta, 0 RGB difference) | +| First versus last frame difference | exactly 0 | +| Video check | 512×640 H.264, 30 fps, 13 frames, 0.433333 seconds | + +Rev2D's documented `eyes_half` variant intentionally remains fully visible over `EyeOpen` 0.3–0.6, explaining the two flat intervals; the tool cannot decide whether this artistic hold is acceptable. **Actual visual inspection** showed eye-outline ghosting at intermediate fade frames, even though the generic jump detector did not flag them. A smooth-looking statistic therefore does not imply smooth-looking animation. The generated variant art is experimental, locally ignored, not published and **not approved for use**. + +## Imported-avatar 25-frame blink review + +For provenance-tracked single-pose Rev2D imports, `node dist/cli/index.js character qa blink avatars/portrait` renders 13 genuine eyelid poses and reuses them in reverse order for a 25-frame montage. The tool saves `blink-eyes.png`, `blink-full.png`, the individual frames and a JSON motion-triage report in a new ignored `outputs/qa/blink-review/` directory. Unlike the general-purpose sequence command, this probes the short-fade boundaries `EyeOpen=0.18/0.19/0.20` and `0.68/0.69/0.70` explicitly. Inspect the montage and source drawings visually; no change to model, assets or approval is made. See [CHARACTER_REVIEW.md](CHARACTER_REVIEW.md). + +A live Hana experiment produced a 25-frame set from 13 engine renders and revealed 10 control-motion-stall candidates in the sampled eye crop, as well as visible double contours and unnatural half-closed eyelids. It remains **unapproved**. Those candidate counts reflect intentional sprite holds as well as potential defects; they are not an automatic accept/reject rating. + +## Limitations and next checks + +- A different or badly deformed eye can produce a *higher* difference score; there is no universal minimum or maximum that certifies good animation. +- Closed-eyelid anatomy, visible tears, head/neck continuity, seam quality, motion cadence, and voice sync still require inspecting rendered frames/video. +- Frame-to-frame cuts, camera movement and background replacement must be identified semantically before interpreting pixel deltas. +- Thresholds should be calibrated per rig on diverse properly licensed artwork, not tuned to one Hana sample. +- This helper does not transmit artwork to an external model. It decodes PNGs locally using FFmpeg. diff --git a/docs/QUALITY_AUDIT.md b/docs/QUALITY_AUDIT.md index ef4fa9b..5ff0d65 100644 --- a/docs/QUALITY_AUDIT.md +++ b/docs/QUALITY_AUDIT.md @@ -73,10 +73,12 @@ Rev2D creates eye-white, iris, upper/lower lid, lash, crease and foreground-hair ## 6. Next milestones -1. Wrap Rev2D drawn-variant requests and imports as guarded PuppetControl CLI operations. Require explicit user/artwork permission and renewed review. No new mesh engine. -2. Add image-based QA probes for rest-pose identity, blink/gaze/mouth activity, and head/neck discontinuities. Quantitative reports are triage, not automatic visual approval. -3. Build a diverse rights-cleared benchmark matrix: eye size, hair over face, accessories/glasses, different skin tones and line styles, opaque versus transparent backgrounds and constrained 3/4 views. Track success/failure on each, not just demos. -4. Only after a reliable single actor: improve body-pose transitions, synchronized multi-actor shots and user-facing authoring polish. +1. **Guarded local drawn-variant CLI:** Rev2D requests masks and accepts explicitly selected, reviewed eye/mouth drawings. The throwaway Hana example was accepted geometrically, but the artwork had flat lids, residual lower contours and doubled fades. Its rig remains **unapproved**. +2. **Experimental short-fade comparison:** Narrower closed→half and half→mesh opacity blends reduced prolonged double contours in selected frames at the cost of faster visible sprite changes. This does not repair skin-tone seams or missing intermediate art. +3. **Bounded PNG QA:** Same-sized rest/activity comparisons, selected eye-region crops and transparency checks provide reproducible diagnostics, never artistic approval. Hana rest/closed eye crop previously showed 33.5879/255 mean RGB error (36.4392% changed). +4. **Ordered sequence and blink montage:** 2–60 ordered PNG frame motion triage flags changed-control stalls and candidate jumps. The dedicated imported-avatar blink command renders 13 engine states and assembles a 25-frame contact sheet including EyeOpen 0.18/0.19/0.20 and 0.68/0.69/0.70. Actual Hana images still show bad half-closed artwork, ghosting and hard seams; the scene is not visually approved. +5. **Rights-cleared multi-character quality benchmark:** Add at least ten different visual styles (eyes, hair occlusion, glasses, color and 3/4 views), track failures instead of extrapolating from one Hana illustration. +6. **After reliable single-actor quality:** Improve head/neck seam checks, genuinely smooth body transitions and multi-actor shots. ### Local reproduction diff --git a/src/audio/cue-provenance.ts b/src/audio/cue-provenance.ts new file mode 100644 index 0000000..d2ef56d --- /dev/null +++ b/src/audio/cue-provenance.ts @@ -0,0 +1,226 @@ +import { createHash } from "node:crypto"; +import { createReadStream } from "node:fs"; +import { open, readFile, realpath, stat } from "node:fs/promises"; +import { basename, dirname, join, relative, resolve, sep } from "node:path"; +import { z } from "zod"; +import { checkedPath } from "../character/avatar.js"; +import { parseRhubarbMouthCuesJson } from "./rhubarb.js"; + +export const CUE_BINDING_FORMAT = "puppetcontrol-rhubarb-cue-binding"; +export const CUE_BINDING_VERSION = 1; + +const MAX_AUDIO_BYTES = 200 * 1024 * 1024; +const MAX_CUES_BYTES = 5 * 1024 * 1024; +const MAX_SIDECAR_BYTES = 64 * 1024; +const SHA256 = /^[a-f0-9]{64}$/; + +const canonicalPathSchema = z.string().min(1).max(1024).refine((value) => { + if (value.includes("\\") || value.includes(":") || value.startsWith("/")) return false; + return !value.split("/").some((segment) => !segment || segment === "." || segment === ".."); +}, "Expected a canonical workspace-relative path"); + +const fileBindingSchema = z.strictObject({ + path: canonicalPathSchema, + sha256: z.string().regex(SHA256), +}); + +export const cueBindingSchema = z.strictObject({ + format: z.literal(CUE_BINDING_FORMAT), + version: z.literal(CUE_BINDING_VERSION), + hashAlgorithm: z.literal("sha256"), + sidecarPath: canonicalPathSchema, + audio: fileBindingSchema, + cues: fileBindingSchema, +}); + +export type CueBinding = z.infer; + +export type CueBindingVerification = + | { valid: true; binding: CueBinding } + | { + valid: false; + code: "CUE_BINDING_MISSING_OR_INVALID" | "CUE_BINDING_STALE"; + message: string; + }; + +export class CueBindingError extends Error { + constructor(public readonly code: string, message: string) { + super(message); + this.name = "CueBindingError"; + } +} + +interface WorkspaceFile { + fullPath: string; + canonicalPath: string; +} + +function relativeCanonical(rootReal: string, fullPath: string): string { + const rel = relative(rootReal, fullPath); + if (!rel || rel === ".." || rel.startsWith(".." + sep)) { + throw new CueBindingError("CUE_BINDING_PATH_INVALID", "Cue binding files must stay inside the workspace"); + } + return rel.replaceAll("\\", "/"); +} + +async function existingWorkspaceFile( + rootReal: string, + input: string, + label: string, + maxBytes: number, +): Promise { + const lexicalPath = await checkedPath(rootReal, resolve(rootReal, input)); + const fullPath = await realpath(lexicalPath); + await checkedPath(rootReal, fullPath); + const info = await stat(fullPath); + if (!info.isFile() || info.size < 1 || info.size > maxBytes) { + throw new CueBindingError( + "CUE_BINDING_INPUT_INVALID", + label + " must be a nonempty regular file no larger than " + maxBytes + " bytes", + ); + } + return { fullPath, canonicalPath: relativeCanonical(rootReal, fullPath) }; +} + +async function newSidecarPath(rootReal: string, input: string): Promise { + const lexicalPath = await checkedPath(rootReal, resolve(rootReal, input)); + const parentReal = await realpath(dirname(lexicalPath)); + await checkedPath(rootReal, parentReal); + const fullPath = join(parentReal, basename(lexicalPath)); + await checkedPath(rootReal, fullPath); + return { fullPath, canonicalPath: relativeCanonical(rootReal, fullPath) }; +} + +async function existingSidecarPath(rootReal: string, input: string): Promise { + return existingWorkspaceFile(rootReal, input, "Cue binding sidecar", MAX_SIDECAR_BYTES); +} + +async function fingerprint(path: string): Promise { + const hash = createHash("sha256"); + for await (const chunk of createReadStream(path)) hash.update(chunk as Buffer); + return hash.digest("hex"); +} + +async function resolveInputs(audioFile: string, cuesFile: string) { + const rootReal = await realpath(process.cwd()); + const [audio, cues] = await Promise.all([ + existingWorkspaceFile(rootReal, audioFile, "Audio file", MAX_AUDIO_BYTES), + existingWorkspaceFile(rootReal, cuesFile, "Rhubarb cues file", MAX_CUES_BYTES), + ]); + if (audio.fullPath === cues.fullPath) { + throw new CueBindingError("CUE_BINDING_INPUT_INVALID", "Audio and Rhubarb cues must be different files"); + } + return { rootReal, audio, cues }; +} + +/** + * Create a deterministic integrity sidecar for one already-aligned Rhubarb cue file. + * + * The record contains only workspace-relative canonical paths and SHA-256 hashes. + * It does not prove who created/reviewed the cues and does not generate lip-sync data. + * Existing sidecars are never overwritten: rebinding must be an explicit caller action. + */ +export async function createCueBinding( + audioFile: string, + cuesFile: string, + sidecarFile: string, +): Promise { + const { rootReal, audio, cues } = await resolveInputs(audioFile, cuesFile); + const sidecar = await newSidecarPath(rootReal, sidecarFile); + if (sidecar.fullPath === audio.fullPath || sidecar.fullPath === cues.fullPath) { + throw new CueBindingError("CUE_BINDING_PATH_INVALID", "Cue binding sidecar must be a separate file"); + } + + const cueText = await readFile(cues.fullPath, "utf8"); + const parsedCues = parseRhubarbMouthCuesJson(cueText); + if (parsedCues.length === 0) { + throw new CueBindingError("CUE_BINDING_INPUT_INVALID", "Rhubarb cue file must contain at least one mouth cue"); + } + const [audioSha256, cuesSha256] = await Promise.all([ + fingerprint(audio.fullPath), + fingerprint(cues.fullPath), + ]); + const binding = cueBindingSchema.parse({ + format: CUE_BINDING_FORMAT, + version: CUE_BINDING_VERSION, + hashAlgorithm: "sha256", + sidecarPath: sidecar.canonicalPath, + audio: { path: audio.canonicalPath, sha256: audioSha256 }, + cues: { path: cues.canonicalPath, sha256: cuesSha256 }, + }); + + let handle; + try { + handle = await open(sidecar.fullPath, "wx"); + await handle.writeFile(JSON.stringify(binding, null, 2) + "\n", "utf8"); + await handle.sync(); + } catch (error) { + if (error instanceof Error && "code" in error && error.code === "EEXIST") { + throw new CueBindingError( + "CUE_BINDING_EXISTS", + "Cue binding sidecar already exists; verify it instead of silently rebinding changed inputs", + ); + } + throw error; + } finally { + await handle?.close(); + } + + return binding; +} + +/** + * Verify that the sidecar still refers to these exact canonical files and exact bytes. + * Missing, malformed, moved, or modified files fail closed and never read paths from + * untrusted sidecar metadata. + */ +export async function verifyCueBinding( + audioFile: string, + cuesFile: string, + sidecarFile: string, +): Promise { + try { + const { rootReal, audio, cues } = await resolveInputs(audioFile, cuesFile); + const sidecar = await existingSidecarPath(rootReal, sidecarFile); + if (sidecar.fullPath === audio.fullPath || sidecar.fullPath === cues.fullPath) { + return { + valid: false, + code: "CUE_BINDING_MISSING_OR_INVALID", + message: "Cue binding sidecar must be separate from the audio and cue files", + }; + } + + const raw = await readFile(sidecar.fullPath, "utf8"); + const binding = cueBindingSchema.parse(JSON.parse(raw) as unknown); + if ( + binding.sidecarPath !== sidecar.canonicalPath || + binding.audio.path !== audio.canonicalPath || + binding.cues.path !== cues.canonicalPath + ) { + return { + valid: false, + code: "CUE_BINDING_STALE", + message: "Cue binding canonical paths no longer match the requested audio, cues, or sidecar", + }; + } + + const [audioSha256, cuesSha256] = await Promise.all([ + fingerprint(audio.fullPath), + fingerprint(cues.fullPath), + ]); + if (binding.audio.sha256 !== audioSha256 || binding.cues.sha256 !== cuesSha256) { + return { + valid: false, + code: "CUE_BINDING_STALE", + message: "Audio or Rhubarb cue bytes changed after the cue binding was created", + }; + } + return { valid: true, binding }; + } catch (error) { + return { + valid: false, + code: "CUE_BINDING_MISSING_OR_INVALID", + message: error instanceof Error ? error.message : String(error), + }; + } +} diff --git a/src/character/avatar.ts b/src/character/avatar.ts index c591a0c..13b9988 100644 --- a/src/character/avatar.ts +++ b/src/character/avatar.ts @@ -148,7 +148,12 @@ export async function runRev2D(repo: string, args: string[], timeout = 120_000): // Return the upstream validator's precise diagnostics rather than hiding them. let output: CommandOutput = {}; try { output = JSON.parse(failure.stdout ?? "{}") as CommandOutput; } catch { /* not JSON */ } - const diagnosticText = output.diagnostics ? JSON.stringify(output.diagnostics).slice(0, 3500) : ""; + const rejected = Array.isArray(output.checks) ? output.checks + .filter((value) => !!value && typeof value === "object" && (value as { ok?: boolean }).ok === false) + .map((value) => ({ id: value.id, code: value.code, reason: value.reason, hint: value.hint })) : []; + const diagnosticText = Array.isArray(output.diagnostics) && output.diagnostics.length + ? JSON.stringify(output.diagnostics).slice(0, 3500) + : rejected.length ? JSON.stringify(rejected).slice(0, 3500) : ""; throw new AvatarWorkflowError( "REV2D_AVATAR_FAILED", [diagnosticText, failure.stderr, failure.message].filter(Boolean).join("\n").slice(0, 5000), diff --git a/src/character/blink-review.ts b/src/character/blink-review.ts new file mode 100644 index 0000000..4103459 --- /dev/null +++ b/src/character/blink-review.ts @@ -0,0 +1,181 @@ +/** + * Generate a visual blink inspection package from one VERIFIED, unapproved or + * approved imported Rev2D avatar. This is not a rig fix or an approval gate. + * + * Closely sample both sprite transitions; generic 4-column Rev2D review + * sheets can otherwise miss double outlines at EyeOpen=0.19 and 0.69. + */ +import { execFile } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { copyFile, mkdir, readFile, stat, writeFile } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; +import { promisify } from "node:util"; +import { characterSchema } from "../scene/schemas.js"; +import { AvatarWorkflowError, checkedPath, pinnedRev2D, runRev2D } from "./avatar.js"; +import { verifyAvatarBuildRecord, checkAvatarPoseCoverage } from "./provenance.js"; +import { comparePngSequence } from "./frame-sequence-qa.js"; +import { parsePixelRegionArgument, type PixelRegion } from "./pixel-qa.js"; + +const execFileAsync = promisify(execFile); +const MAX_MODEL_BYTES = 20_000_000; +export const BLINK_CLOSING_VALUES = [ + 1, 0.83, 0.71, 0.70, 0.69, 0.68, 0.50, + 0.21, 0.20, 0.19, 0.18, 0.17, 0, +] as const; +/** 25 evenly-timed frame slots sampled from 13 explicitly rendered rig states. */ +export const BLINK_REVIEW_VALUES: number[] = [ + ...BLINK_CLOSING_VALUES, + ...BLINK_CLOSING_VALUES.slice(0, -1).reverse(), +]; + +type RecordValue = Record; +function record(value: unknown): RecordValue | null { + return value !== null && typeof value === "object" && !Array.isArray(value) + ? value as RecordValue : null; +} + +function openingBounds(value: unknown): { left: number; right: number; top: number; bottom: number } | null { + if (!Array.isArray(value) || value.length < 8 || value.length % 2) return null; + const xs: number[] = []; + const ys: number[] = []; + for (let i = 0; i < value.length; i += 2) { + const x = value[i]; const y = value[i + 1]; + if (!Number.isFinite(x) || !Number.isFinite(y)) return null; + xs.push(x as number); ys.push(y as number); + } + return { left: Math.min(...xs), right: Math.max(...xs), top: Math.min(...ys), bottom: Math.max(...ys) }; +} + +/** Eyes ROI is a starting crop for human inspection, not an accuracy rating. */ +export function eyeRegionFromAvatarSpec( + spec: unknown, width: number, height: number, +): PixelRegion { + const face = record(record(spec)?.face); + const eyes = record(face?.eyes); + const bounds = ["imageLeft", "imageRight"] + .map((side) => openingBounds(record(eyes?.[side])?.opening)); + if (bounds.some((item) => item === null) || !Number.isSafeInteger(width) || + !Number.isSafeInteger(height) || width < 256 || height < 256) { + throw new AvatarWorkflowError("BLINK_EYE_REGION_REQUIRED", + "Cannot infer both eye openings from avatar.json. Provide --region x,y,width,height."); + } + const both = bounds as NonNullable[]; + const left = Math.max(0, Math.floor(Math.min(...both.map((b) => b.left)) - 26)); + const top = Math.max(0, Math.floor(Math.min(...both.map((b) => b.top)) - 24)); + const right = Math.min(width, Math.ceil(Math.max(...both.map((b) => b.right)) + 26)); + const bottom = Math.min(height, Math.ceil(Math.max(...both.map((b) => b.bottom)) + 24)); + if (right <= left || bottom <= top) { + throw new AvatarWorkflowError("BLINK_EYE_REGION_REQUIRED", "Invalid eye-opening crop."); + } + return { x: left, y: top, width: right - left, height: bottom - top }; +} + +async function contactSheet( + framesDirectory: string, outputFile: string, + crop?: PixelRegion, +): Promise { + const cropStage = crop ? `crop=${crop.width}:${crop.height}:${crop.x}:${crop.y},` : ""; + const scaleStage = crop ? "scale=490:-2:flags=lanczos" : "scale=256:-2:flags=lanczos"; + try { + await execFileAsync("ffmpeg", [ + "-nostdin", "-hide_banner", "-loglevel", "error", "-y", + "-framerate", "30", "-i", join(framesDirectory, "frame-%03d.png"), + "-vf", cropStage + scaleStage + ",tile=5x5", "-frames:v", "1", outputFile, + ], { windowsHide: true, shell: false, timeout: 90_000, maxBuffer: 4_000_000 }); + } catch (cause) { + throw new AvatarWorkflowError("BLINK_CONTACT_FAILED", + "Cannot create blink contact sheet with FFmpeg: " + String(cause)); + } +} + +export async function createBlinkReview( + projectDir: string, options: { region?: PixelRegion; root?: string } = {}, +) { + const root = resolve(options.root ?? process.cwd()); + const project = await checkedPath(root, resolve(root, projectDir)); + const characterFile = await checkedPath(project, join(project, "character.json")); + const manifest = characterSchema.parse(JSON.parse(await readFile(characterFile, "utf8")) as unknown); + if (!manifest.provenance || !manifest.render || + manifest.poseFamilies.length !== 1) { + throw new AvatarWorkflowError("BLINK_REVIEW_IMPORTED_AVATAR_REQUIRED", + "Blink review requires one provenance-tracked imported Rev2D actor, with or without approval."); + } + const coverage = checkAvatarPoseCoverage(manifest); + if (!coverage.valid) throw new AvatarWorkflowError(coverage.code, coverage.message); + const provenance = await verifyAvatarBuildRecord(characterFile, manifest.render.modelPath); + if (!provenance.valid) throw new AvatarWorkflowError(provenance.code, provenance.message); + const modelPath = await checkedPath(project, resolve(project, manifest.render.modelPath)); + const stats = await stat(modelPath); + if (!stats.isFile() || stats.size > MAX_MODEL_BYTES) { + throw new AvatarWorkflowError("BLINK_REVIEW_MODEL_INVALID", "Missing/oversized rig model."); + } + const model = JSON.parse(await readFile(modelPath, "utf8")) as { + meta?: { width: number; height: number }; + parameters?: { id: string; min: number; max: number }[]; + }; + const width = model.meta?.width; + const height = model.meta?.height; + if (!Number.isSafeInteger(width) || !Number.isSafeInteger(height) || + width! < 256 || height! < 256 || width! > 4096 || height! > 4096 || + !["EyeLOpen", "EyeROpen"].every((name) => + model.parameters?.some((parameter) => + parameter.id === name && parameter.min <= 0 && parameter.max >= 1))) { + throw new AvatarWorkflowError("BLINK_REVIEW_EYE_CONTROLS_MISSING", + "Rig requires EyeLOpen and EyeROpen parameters accepting 0–1."); + } + const spec = JSON.parse(await readFile(join(project, "avatar.json"), "utf8")) as unknown; + const region = options.region ?? eyeRegionFromAvatarSpec(spec, width!, height!); + // Reuse canonical region validation in the bounded PNG comparator. + parsePixelRegionArgument(`${region.x},${region.y},${region.width},${region.height}`); + if (region.x + region.width > width! || region.y + region.height > height!) { + throw new AvatarWorkflowError("BLINK_EYE_REGION_REQUIRED", "Eye region exceeds the rig canvas."); + } + const backend = await pinnedRev2D(root); + const runRoot = await checkedPath(root, join(root, "outputs", "qa", "blink-review", randomUUID())); + const framesDir = join(runRoot, "frames"); + await mkdir(framesDir, { recursive: true }); + + const frames: string[] = []; + for (let index = 0; index < BLINK_CLOSING_VALUES.length; index++) { + const value = BLINK_CLOSING_VALUES[index]!; + const path = join(framesDir, `frame-${String(index).padStart(3, "0")}.png`); + await runRev2D(backend, [ + "render", modelPath, "-o", path, "--param", `EyeLOpen=${value}`, + "--param", `EyeROpen=${value}`, "--bg", "#f0eaf4", "--ss", "1", + ], 60_000); + frames.push(path); + } + for (let index = BLINK_CLOSING_VALUES.length; index < BLINK_REVIEW_VALUES.length; index++) { + const sourceIndex = BLINK_REVIEW_VALUES.length - 1 - index; + const path = join(framesDir, `frame-${String(index).padStart(3, "0")}.png`); + await copyFile(frames[sourceIndex]!, path); + frames.push(path); + } + const report = await comparePngSequence(frames, { + region, controlValues: BLINK_REVIEW_VALUES, + }, root); + const fullSheet = join(runRoot, "blink-full.png"); + const eyeSheet = join(runRoot, "blink-eyes.png"); + await contactSheet(framesDir, fullSheet); + await contactSheet(framesDir, eyeSheet, region); + const reportPath = join(runRoot, "blink-review.json"); + const evidence = { + version: 1, + status: "visual-review-required", + approvedByThisCommand: false, + projectDir: project, + values: BLINK_REVIEW_VALUES, + closingFramesRendered: BLINK_CLOSING_VALUES.length, + openingFramesReused: BLINK_REVIEW_VALUES.length - BLINK_CLOSING_VALUES.length, + fpsForContactOrdering: 30, + region, report, + fullSheet, eyeSheet, framesDirectory: framesDir, + notes: [ + "Static eye sprites can plateau despite control changes; inspect every contact panel, particularly EyeOpen=0.19/0.69.", + "Numbers cannot assess lash seams, iris clipping, head turns or whether eyelids look natural.", + "This command never changes the rig or its approval state. A human must still approve intentionally.", + ], + }; + await writeFile(reportPath, JSON.stringify(evidence, null, 2) + "\n", { flag: "wx" }); + return { valid: true, ...evidence, reportPath }; +} diff --git a/src/character/drawn-variants.ts b/src/character/drawn-variants.ts new file mode 100644 index 0000000..ba07a9d --- /dev/null +++ b/src/character/drawn-variants.ts @@ -0,0 +1,202 @@ +/** + * Guarded local artwork variant workflow for imported Rev2D avatars. + * + * The generated image itself is never sent to an AI provider by these calls. + * A successful import mutates the model/textures, invalidates previous visual + * approval, rebuilds review sheets, then refreshes exact asset fingerprints. + */ +import { open, readFile, rename, stat, unlink, writeFile } from "node:fs/promises"; +import { randomUUID } from "node:crypto"; +import { dirname, join } from "node:path"; +import { characterSchema } from "../scene/schemas.js"; +import { shortFadeDrawnEyes } from "./eye-short-fade.js"; +import { + AvatarWorkflowError, checkedPath, parsePngSize, pinnedRev2D, runRev2D, +} from "./avatar.js"; +import { + createAvatarBuildRecord, revokeAvatarApprovalBeforeBuild, + verifyAvatarBuildRecord, writeAvatarBuildRecord, +} from "./provenance.js"; + +export const DRAWN_VARIANT_IDS = [ + "eyes_closed", "eyes_half", "eyes_smile", + "mouth_a", "mouth_a_half", "mouth_i", "mouth_u", "mouth_e", "mouth_o", +] as const; + +const ALLOWED_VARIANTS = new Set(DRAWN_VARIANT_IDS); +const DEFAULT_VARIANTS = ["eyes_closed", "mouth_a"]; +const MAX_DRAWING_BYTES = 30 * 1024 * 1024; +const MAX_MODEL_BYTES = 20 * 1024 * 1024; + +export function validateDrawnVariantIds( + requested: readonly string[], requireExplicit = false, +): string[] { + if (requireExplicit && requested.length === 0) { + throw new AvatarWorkflowError( + "VARIANT_IDS_REQUIRED", + "Select the exact reviewed artwork IDs to import; bulk importing all drawings is not allowed.", + ); + } + if (requested.length > DRAWN_VARIANT_IDS.length) { + throw new AvatarWorkflowError("VARIANT_IDS_INVALID", "Too many drawn variant IDs."); + } + const ids = requested.length ? requested : DEFAULT_VARIANTS; + if (ids.some((id) => !ALLOWED_VARIANTS.has(id)) || new Set(ids).size !== ids.length) { + throw new AvatarWorkflowError( + "VARIANT_IDS_INVALID", + "Unknown or duplicate variant ID. Supported: " + DRAWN_VARIANT_IDS.join(", "), + ); + } + return [...ids]; +} + +async function projectPaths(projectDir: string, root: string) { + const project = await checkedPath(root, projectDir); + if (!(await stat(project)).isDirectory()) { + throw new AvatarWorkflowError("AVATAR_PROJECT_REQUIRED", "Expected a built avatar project directory."); + } + return { + project, + manifestPath: await checkedPath(root, join(project, "character.json")), + }; +} + +async function verifiedImportedAvatar(projectDir: string, root: string) { + const paths = await projectPaths(projectDir, root); + const raw = await readFile(paths.manifestPath, "utf8"); + const manifest = characterSchema.parse(JSON.parse(raw) as unknown); + if (!manifest.provenance || !manifest.render?.modelPath || + manifest.poseFamilies.length !== 1 || + manifest.poseFamilies[0]?.modelPath !== manifest.render.modelPath) { + throw new AvatarWorkflowError( + "AVATAR_VARIANTS_UNSUPPORTED", + "Drawn variant import currently requires a single-pose character built with puppetctl character build.", + ); + } + const integrity = await verifyAvatarBuildRecord(paths.manifestPath, manifest.render.modelPath); + if (!integrity.valid) { + throw new AvatarWorkflowError(integrity.code, + "Avatar source/rig assets are stale: " + integrity.message + + ". Restore the verified build before modifying drawn variants."); + } + return { ...paths, manifest }; +} + +async function readDrawingSize(path: string): Promise<{ width: number; height: number }> { + const stats = await stat(path); + if (!stats.isFile() || stats.size < 24 || stats.size > MAX_DRAWING_BYTES) { + throw new AvatarWorkflowError("VARIANT_DRAWING_INVALID", "Drawing must be a PNG file under 30 MB."); + } + const handle = await open(path, "r"); + try { + const buffer = Buffer.alloc(24); + if ((await handle.read(buffer, 0, buffer.length, 0)).bytesRead !== buffer.length) { + throw new AvatarWorkflowError("VARIANT_DRAWING_INVALID", "Incomplete PNG header."); + } + return parsePngSize(buffer); + } finally { + await handle.close(); + } +} + +/** Prepare masks and prompts only; no generated texture or approval changes. */ +export async function requestDrawnVariants( + projectDir: string, requested: readonly string[] = [], root = process.cwd(), +) { + const ids = validateDrawnVariantIds(requested); + const { project } = await verifiedImportedAvatar(projectDir, root); + const backend = await pinnedRev2D(root); + const output: Record = await runRev2D(backend, ["avatar", "variants", "request", project, ...ids]); + return { + valid: true, status: "artwork-required", projectDir: project, variants: ids, + masks: ids.map((id) => join(project, "variant-requests", id + ".mask.png")), + prompts: ids.map((id) => join(project, "variant-requests", id + ".prompt.txt")), + backend: output, + next: "Draw full-size copies of source.png, editing ONLY the mask's transparent region. Save to variants/.png. Check image rights; these commands never upload artwork.", + }; +} + +/** Report backend state, including rejected/stale variants, even during recovery. */ +export async function statusDrawnVariants(projectDir: string, root = process.cwd()): Promise> { + const { project } = await projectPaths(projectDir, root); + const backend = await pinnedRev2D(root); + return runRev2D(backend, ["avatar", "variants", "status", project]); +} + +/** + * Adjust only derived eye-opacity curves of a locally reviewed rig. + * The calling import operation revokes visual approval before writing. + * Validation and new fingerprints are required after this step. + */ +async function applyExperimentalShortFade( + project: string, modelRelative: string, backend: string, +) { + const modelPath = await checkedPath(project, join(project, modelRelative)); + const file = await stat(modelPath); + if (!file.isFile() || file.size > MAX_MODEL_BYTES) { + throw new AvatarWorkflowError("EYE_SHORT_FADE_UNSUPPORTED", "Invalid or oversized Rev2D model."); + } + const transformed = shortFadeDrawnEyes(JSON.parse(await readFile(modelPath, "utf8")) as unknown); + const temporary = join(dirname(modelPath), ".puppetcontrol-shortfade-" + randomUUID() + ".r2d.json"); + try { + await writeFile(temporary, JSON.stringify(transformed.model), { flag: "wx" }); + await runRev2D(backend, ["validate", temporary]); + await rename(temporary, modelPath); + } finally { + await unlink(temporary).catch(() => undefined); + } + return { profile: "experimental-short-fade" as const, ...transformed.profile, changedBindings: transformed.changedBindings }; +} +/** Import reviewed artwork, leave unapproved; preserve stale hashes on a partial failure. */ +export async function importDrawnVariants( + projectDir: string, requested: readonly string[], root = process.cwd(), + options: { experimentalShortFade?: boolean } = {}, +) { + const ids = validateDrawnVariantIds(requested, true); + const { project, manifestPath, manifest } = await verifiedImportedAvatar(projectDir, root); + const source = await checkedPath(root, join(project, "source.png")); + const size = await readDrawingSize(source); + for (const id of ids) { + const drawing = await checkedPath(root, join(project, "variants", id + ".png")); + const drawnSize = await readDrawingSize(drawing); + if (size.width !== drawnSize.width || size.height !== drawnSize.height) { + throw new AvatarWorkflowError( + "VARIANT_SIZE_MISMATCH", + id + " must have exactly the source illustration's dimensions.", + ); + } + } + + const backend = await pinnedRev2D(root); + // IMPORTANT: revoke before Rev2D can touch model, layers or texture files. + await revokeAvatarApprovalBeforeBuild(manifestPath); + let imported: unknown; + let profile: Awaited> | undefined; + try { + imported = await runRev2D(backend, ["avatar", "variants", "import", project, ...ids], 240_000); + if (options.experimentalShortFade) { + profile = await applyExperimentalShortFade(project, manifest.render!.modelPath, backend); + } + await runRev2D(backend, ["avatar", "poses", project], 240_000); + const record = await createAvatarBuildRecord(project, manifest.render!.modelPath); + await writeAvatarBuildRecord(project, record); + } catch (error) { + throw new AvatarWorkflowError( + "VARIANT_IMPORT_REVIEW_REQUIRED", + (error instanceof Error ? error.message : String(error)) + + "\nThe avatar remains unapproved. Rev2D may have partially imported artwork; " + + "if fingerprints are now stale, rebuild with character build --force before retrying.", + ); + } + return { + valid: true, status: "review-required", approved: false, projectDir: project, + variants: ids, backend: imported, + ...(profile ? { eyeTransition: profile } : {}), + review: { + eyes: join(project, "review", "eyes.png"), + mouth: join(project, "review", "mouth.png"), + poses: join(project, "review", "poses.png"), + }, + next: "Inspect the newly rendered review sheets for seams, color changes and facial jumps. Reapprove character.json explicitly only after visual QA.", + }; +} diff --git a/src/character/eye-short-fade.ts b/src/character/eye-short-fade.ts new file mode 100644 index 0000000..1cad0fe --- /dev/null +++ b/src/character/eye-short-fade.ts @@ -0,0 +1,148 @@ +/** + * Experimental short-fade profile for Rev2D's *drawn* half/closed eye sprites. + * + * The upstream engine linearly dissolves two complete eye illustrations across + * EyeOpen 0.12–0.2 and 0.6–0.7. Double lash lines become visible when their + * geometry differs. Narrowing the opacity windows reduces how long both drawings + * overlap, but deliberately trades smoothness for crispness. + * + * Never apply to a source or approved model in place. Callers must revoke visual + * approval, validate the derived model and regenerate review images and hashes. + */ +import { AvatarWorkflowError } from "./avatar.js"; + +type JsonRecord = Record; +type Rev2DBinding = { + id: string; + target: string; + params: string[]; + keys: Array<{ at: number[]; value: number }>; +}; + +export const SHORT_FADE_EYE_PROFILE = { + /** Fade from closed to half across this EyeOpen range. */ + closedToHalf: [0.18, 0.20], + /** Fade from half to mesh across this EyeOpen range. */ + halfToMesh: [0.68, 0.70], +} as const; + +export interface ShortFadeResult { + /** A NEW model object. Caller owns its publication and renewed review. */ + model: JsonRecord; + changedBindings: string[]; + profile: typeof SHORT_FADE_EYE_PROFILE; +} + +function unsupported(why: string): never { + throw new AvatarWorkflowError( + "EYE_SHORT_FADE_UNSUPPORTED", + "Cannot safely apply the optional short-fade policy: " + why + + ". Leave the upstream rig unchanged and inspect its review sheets.", + ); +} + +function isBinding(input: unknown): input is Rev2DBinding { + if (!input || typeof input !== "object" || Array.isArray(input)) return false; + const b = input as Partial; + return typeof b.id === "string" && + typeof b.target === "string" && + Array.isArray(b.params) && b.params.every((p) => typeof p === "string") && + Array.isArray(b.keys) && b.keys.length >= 2 && b.keys.every( + (key) => key && Array.isArray(key.at) && key.at.length === 1 && + typeof key.at[0] === "number" && Number.isFinite(key.at[0]) && + typeof key.value === "number" && Number.isFinite(key.value), + ); +} + +function points(entries: Array): Rev2DBinding["keys"] { + return entries.map(([at, value]) => ({ at: [at], value })); +} + +const expectedSprites = [ + ["sprite_eyes_closed_imageLeftShow", "EyeROpen"], + ["sprite_eyes_closed_imageRightShow", "EyeLOpen"], + ["sprite_eyes_half_imageLeftShow", "EyeROpen"], + ["sprite_eyes_half_imageRightShow", "EyeLOpen"], +] as const; + +function expectedParam(id: string): string | null { + if (id.includes("imageLeft") || id.startsWith("eyeR_")) return "EyeROpen"; + if (id.includes("imageRight") || id.startsWith("eyeL_")) return "EyeLOpen"; + return null; +} + +/** Pure transform, only on the known, fully supported two-eye sprite topology. */ +export function shortFadeDrawnEyes(input: unknown): ShortFadeResult { + if (!input || typeof input !== "object" || Array.isArray(input)) unsupported("invalid model JSON"); + const candidate = structuredClone(input) as JsonRecord; + const bindings = candidate.bindings; + if (!Array.isArray(bindings)) unsupported("missing bindings"); + // Other Rev2D bindings may control multi-parameter warps and structured + // vertex offsets rather than scalar opacity; preserve them untouched. + const eyeBindings: Rev2DBinding[] = []; + const byId = new Map(); + for (const value of bindings) { + if (!value || typeof value !== "object") continue; + const id = (value as { id?: unknown }).id; + if (typeof id !== "string") continue; + if (byId.has(id)) unsupported("duplicate binding " + id); + if (!isBinding(value)) { + if (id.startsWith("sprite_eyes_") || id.endsWith("UnderSprite")) unsupported("unsupported eye binding " + id); + continue; + } + byId.set(id, value); + eyeBindings.push(value); + } + if ([...byId.keys()].some((id) => id.startsWith("sprite_eyes_smile_"))) { + unsupported("the separate eyes_smile layering has not been checked with this profile"); + } + for (const [id, param] of expectedSprites) { + const binding = byId.get(id); + const partId = id.slice(0, -"Show".length); + if (!binding || binding.target !== "part:" + partId + ".opacity" || + binding.params.length !== 1 || binding.params[0] !== param) { + unsupported("missing or incompatible " + id + "; both reviewed half/closed variants are required"); + } + } + const underneath = eyeBindings.filter((b) => + /^eye[LR]_(?:white|ball|lower|upper|lowerRim|crease|upperRim)UnderSprite$/.test(b.id)); + for (const side of ["eyeL_", "eyeR_"]) { + if (underneath.filter((b) => b.id.startsWith(side)).length < 4) { + unsupported("incomplete eye-mesh hide controls for " + side); + } + } + for (const b of underneath) { + const param = expectedParam(b.id); + if (!param || b.params.length !== 1 || b.params[0] !== param || + b.target !== "part:" + b.id.slice(0, -"UnderSprite".length) + ".opacity") { + unsupported("incompatible eye mesh opacity control " + b.id); + } + const hidden = b.keys[0]!.value; + const visible = b.keys[b.keys.length - 1]!.value; + if (Math.abs(hidden) > 1e-5 || Math.abs(visible - 1) > 1e-5) { + unsupported("nonstandard eye mesh opacity limits for " + b.id); + } + } + + const [closedStart, closedEnd] = SHORT_FADE_EYE_PROFILE.closedToHalf; + const [openStart, openEnd] = SHORT_FADE_EYE_PROFILE.halfToMesh; + const changed: string[] = []; + for (const b of eyeBindings) { + if (b.id.startsWith("sprite_eyes_closed_") && b.id.endsWith("Show")) { + b.keys = points([[0, 1], [closedStart, 1], [closedEnd, 0], [1, 0]]); + } else if (b.id.startsWith("sprite_eyes_half_") && b.id.endsWith("Show")) { + b.keys = points([[0, 0], [closedStart, 0], [closedEnd, 1], + [openStart, 1], [openEnd, 0], [1, 0]]); + } else if (underneath.includes(b)) { + b.keys = points([[0, 0], [openStart, 0], [openEnd, 1], [1, 1]]); + } else { + continue; + } + changed.push(b.id); + } + // Fail closed if an upgraded Rev2D adds a different sprite curve not covered. + if (changed.filter((id) => id.endsWith("Show")).length !== expectedSprites.length) { + unsupported("not every known eye sprite can be transformed"); + } + return { model: candidate, changedBindings: changed, profile: SHORT_FADE_EYE_PROFILE }; +} diff --git a/src/character/frame-sequence-qa.ts b/src/character/frame-sequence-qa.ts new file mode 100644 index 0000000..a5f291e --- /dev/null +++ b/src/character/frame-sequence-qa.ts @@ -0,0 +1,171 @@ +/** + * Time-ordered PNG motion triage. + * + * Adjacent-frame pixel deltas reveal candidates for *manual* jump inspection. + * A legitimate fast blink, mouth accent, camera cut or hard pose edit can yield + * a high score; the report never certifies or rejects animation quality. + */ +import { AvatarWorkflowError } from "./avatar.js"; +import { comparePngFiles, type PixelComparison, type PixelRegion } from "./pixel-qa.js"; + +export const MAX_SEQUENCE_FRAMES = 60; + +export interface SequenceOptions { + region?: PixelRegion; + /** Transition i→i+1 is editorially intentional and not a surprise-jump candidate. */ + cutAfter?: readonly number[]; + /** Optional expected control sampled at each frame; used only for motion-stall triage. */ + controlValues?: readonly number[]; +} + +export interface SequencePair { + from: number; + to: number; + intentionalCut: boolean; + rgbMeanAbsoluteError: number; + changedFraction: number; + unexpectedTransparentBorderPixels: number; + controlDelta: number | null; + /** Triage only. Do not treat false as proof that an animation is smooth. */ + abruptChangeCandidate: boolean; + motionStallCandidate: boolean; +} + +export interface SequenceQaReport { + frameCount: number; + pairCount: number; + region: PixelRegion; + medianPairError: number; + medianAbsoluteDeviation: number; + reviewThreshold: number; + pairs: SequencePair[]; + possibleAbruptChanges: number[]; + possibleMotionStalls: number[]; + controlValuesProvided: boolean; + firstToLast: { + rgbMeanAbsoluteError: number; + changedFraction: number; + unexpectedTransparentBorderPixels: number; + }; + note: string; +} + +function median(values: readonly number[]): number { + const ordered = [...values].sort((a, b) => a - b); + const middle = Math.floor(ordered.length / 2); + return ordered.length % 2 === 1 + ? ordered[middle]! + : (ordered[middle - 1]! + ordered[middle]!) / 2; +} + +function finiteMetric(comparison: PixelComparison): void { + if (!Number.isFinite(comparison.rgbMeanAbsoluteError) || + !Number.isFinite(comparison.changedFraction) || + comparison.rgbMeanAbsoluteError < 0 || comparison.rgbMeanAbsoluteError > 255 || + comparison.changedFraction < 0 || comparison.changedFraction > 1) { + throw new AvatarWorkflowError("SEQUENCE_QA_METRIC_INVALID", "Pixel comparisons must return bounded numeric metrics."); + } +} + +function checkSequenceOptions(frameCount: number, options: SequenceOptions): Set { + if (frameCount < 2 || frameCount > MAX_SEQUENCE_FRAMES || !Number.isSafeInteger(frameCount)) { + throw new AvatarWorkflowError("SEQUENCE_QA_LENGTH_INVALID", "Supply 2–60 ordered frames."); + } + const cuts = options.cutAfter ?? []; + if (options.controlValues && (options.controlValues.length !== frameCount || + options.controlValues.some((value) => !Number.isFinite(value) || Math.abs(value) > 1_000_000))) { + throw new AvatarWorkflowError("SEQUENCE_QA_CONTROL_VALUES_INVALID", + "Control samples must contain one finite value for each frame."); + } + if (cuts.some((i) => !Number.isSafeInteger(i) || i < 0 || i >= frameCount - 1) || + new Set(cuts).size !== cuts.length) { + throw new AvatarWorkflowError("SEQUENCE_QA_CUTS_INVALID", + "Each intentional cut must be a unique zero-based frame index before the last frame."); + } + return new Set(cuts); +} + +function compactMetric(comparison: PixelComparison) { + return { + rgbMeanAbsoluteError: comparison.rgbMeanAbsoluteError, + changedFraction: comparison.changedFraction, + unexpectedTransparentBorderPixels: comparison.unexpectedTransparentBorderPixels, + }; +} + +/** + * Summarize fixed-size image comparisons. Exclude *declared* editorial cuts + * from baseline statistics; don't mislabel them as unexpected pops. + * In a sequence without at least 3 comparable non-cut pairs, classify no + * abrupt-change candidates (the evidence is too sparse). + */ +export function analyzeSequenceComparisons( + pairs: readonly PixelComparison[], firstToLast: PixelComparison, + options: SequenceOptions = {}, +): SequenceQaReport { + const cuts = checkSequenceOptions(pairs.length + 1, options); + for (const comparison of [...pairs, firstToLast]) finiteMetric(comparison); + const dims = pairs[0]!; + const width = dims.width; + const height = dims.height; + const region = dims.region; + const sameRegion = (a: PixelRegion, b: PixelRegion) => + a.x === b.x && a.y === b.y && a.width === b.width && a.height === b.height; + if ([...pairs, firstToLast].some((comparison) => + comparison.width !== width || comparison.height !== height || !sameRegion(comparison.region, region))) { + throw new AvatarWorkflowError("SEQUENCE_QA_DIMENSIONS_MISMATCH", + "Comparisons must share the same exact frame dimensions and selected region."); + } + + const comparable = pairs + .map((comparison, i) => ({ i, score: comparison.rgbMeanAbsoluteError })) + .filter(({ i }) => !cuts.has(i)).map(({ score }) => score); + const center = comparable.length ? median(comparable) : 0; + const mad = comparable.length ? median(comparable.map((score) => Math.abs(score - center))) : 0; + + // Avoid zero-noise score jumps and render compression artifacts. + // Stronger than typical adjacent variation, not a quality certification. + const threshold = Number(Math.min(255, Math.max(8, center * 2.5, center + 6 * mad)).toFixed(4)); + const pairReports: SequencePair[] = pairs.map((comparison, from) => ({ + from, to: from + 1, intentionalCut: cuts.has(from), + ...compactMetric(comparison), + controlDelta: options.controlValues + ? Number(Math.abs(options.controlValues[from + 1]! - options.controlValues[from]!).toFixed(6)) + : null, + motionStallCandidate: !!options.controlValues && + Math.abs(options.controlValues[from + 1]! - options.controlValues[from]!) > 0.000001 && + comparison.rgbMeanAbsoluteError <= 0.05 && comparison.changedFraction <= 0.001, + abruptChangeCandidate: comparable.length >= 3 && !cuts.has(from) && + comparison.rgbMeanAbsoluteError > threshold, + })); + return { + frameCount: pairs.length + 1, + pairCount: pairs.length, + region, + medianPairError: Number(center.toFixed(4)), + medianAbsoluteDeviation: Number(mad.toFixed(4)), + reviewThreshold: threshold, + pairs: pairReports, + possibleAbruptChanges: pairReports.filter((p) => p.abruptChangeCandidate).map((p) => p.from), + possibleMotionStalls: pairReports.filter((p) => p.motionStallCandidate).map((p) => p.from), + controlValuesProvided: !!options.controlValues, + firstToLast: compactMetric(firstToLast), + note: "Jump and changed-control motion-stall candidates are triage, not proof of defects. Inspect frames, rig clamping, artwork visibility, timing and declared cuts before judging smoothness.", + }; +} + +/** File-by-file scan bounds memory usage; all files must stay inside the current workspace. */ +export async function comparePngSequence( + files: readonly string[], options: SequenceOptions = {}, root = process.cwd(), +): Promise { + checkSequenceOptions(files.length, options); + if (files.some((file) => typeof file !== "string" || !file.toLowerCase().endsWith(".png"))) { + throw new AvatarWorkflowError("SEQUENCE_QA_PNG_REQUIRED", "Every explicitly ordered frame must be a PNG."); + } + const pairs: PixelComparison[] = []; + for (let i = 0; i < files.length - 1; i++) { + pairs.push(await comparePngFiles(files[i]!, files[i + 1]!, options.region, root)); + } + const last = await comparePngFiles(files[0]!, files[files.length - 1]!, options.region, root); + return analyzeSequenceComparisons(pairs, last, options); +} diff --git a/src/character/pixel-qa.ts b/src/character/pixel-qa.ts new file mode 100644 index 0000000..0b8f248 --- /dev/null +++ b/src/character/pixel-qa.ts @@ -0,0 +1,190 @@ +/** + * Deterministic pixel-level *triage* for PNG render comparisons. + * + * These metrics establish changed pixels and reveal opaque-canvas gaps. + * They DO NOT measure artistic quality or automatically approve a rig. + */ +import { spawn } from "node:child_process"; +import { open, stat } from "node:fs/promises"; +import { AvatarWorkflowError, checkedPath, parsePngSize } from "./avatar.js"; + +const MAX_PNG_BYTES = 30 * 1024 * 1024; +const MAX_RAW_BYTES = 4096 * 4096 * 4; +const PIXEL_DIFFERENCE_THRESHOLD = 10; + +export interface PixelRegion { + x: number; + y: number; + width: number; + height: number; +} + +export interface PixelComparison { + width: number; + height: number; + region: PixelRegion; + comparedPixels: number; + ignoredTransparentPixels: number; + rgbMeanAbsoluteError: number; + changedPixels: number; + changedFraction: number; + /** Reference-opaque edge pixels that became semi/fully transparent. */ + unexpectedTransparentBorderPixels: number; + note: string; +} + +function validRegion(region: PixelRegion, width: number, height: number): void { + if ([region.x, region.y, region.width, region.height].some((value) => !Number.isSafeInteger(value)) || + region.x < 0 || region.y < 0 || region.width < 1 || region.height < 1 || + region.x + region.width > width || region.y + region.height > height) { + throw new AvatarWorkflowError("PIXEL_QA_REGION_INVALID", "Region must be an integer rectangle within both images."); + } +} + +export function parsePixelRegionArgument(raw: string): PixelRegion { + if (!/^\d+,\d+,\d+,\d+$/.test(raw)) { + throw new AvatarWorkflowError("PIXEL_QA_REGION_INVALID", "Use --region x,y,width,height with nonnegative integer values."); + } + const [x, y, width, height] = raw.split(",").map(Number); + if (![x, y, width, height].every((value) => Number.isSafeInteger(value)) || width! < 1 || height! < 1) { + throw new AvatarWorkflowError("PIXEL_QA_REGION_INVALID", "Region size must be positive and coordinates must be safe integers."); + } + return { x: x!, y: y!, width: width!, height: height! }; +} + +export function compareRgbaPixels( + reference: Uint8Array, candidate: Uint8Array, width: number, height: number, + region: PixelRegion = { x: 0, y: 0, width, height }, +): PixelComparison { + if (!Number.isSafeInteger(width) || !Number.isSafeInteger(height) || + width < 1 || height < 1 || width * height * 4 > MAX_RAW_BYTES || + reference.byteLength !== width * height * 4 || candidate.byteLength !== width * height * 4) { + throw new AvatarWorkflowError("PIXEL_QA_BUFFER_INVALID", "Both images need matching RGBA buffers and valid bounded dimensions."); + } + validRegion(region, width, height); + + let compared = 0; + let skipped = 0; + let rgbAbsoluteSum = 0; + let changed = 0; + for (let y = region.y; y < region.y + region.height; y++) { + for (let x = region.x; x < region.x + region.width; x++) { + const i = (y * width + x) * 4; + const refAlpha = reference[i + 3]!; + const nextAlpha = candidate[i + 3]!; + // RGB is undefined/invisible when both samples are completely transparent. + if (refAlpha < 16 && nextAlpha < 16) { + skipped++; + continue; + } + let maxChannelDelta = 0; + for (let channel = 0; channel < 3; channel++) { + const delta = Math.abs(reference[i + channel]! - candidate[i + channel]!); + rgbAbsoluteSum += delta; + maxChannelDelta = Math.max(maxChannelDelta, delta); + } + if (maxChannelDelta > PIXEL_DIFFERENCE_THRESHOLD || + Math.abs(refAlpha - nextAlpha) > PIXEL_DIFFERENCE_THRESHOLD) changed++; + compared++; + } + } + + let unexpectedTransparentBorderPixels = 0; + // Count each perimeter pixel once (avoid counting corners twice). + for (let y = 0; y < height; y++) { + for (const x of (width === 1 ? [0] : [0, width - 1])) { + const i = (y * width + x) * 4 + 3; + if (reference[i]! >= 250 && candidate[i]! < 250) unexpectedTransparentBorderPixels++; + } + } + for (let x = 1; x < width - 1; x++) { + for (const y of (height === 1 ? [0] : [0, height - 1])) { + const i = (y * width + x) * 4 + 3; + if (reference[i]! >= 250 && candidate[i]! < 250) unexpectedTransparentBorderPixels++; + } + } + + return { + width, height, region, comparedPixels: compared, ignoredTransparentPixels: skipped, + rgbMeanAbsoluteError: compared ? Number((rgbAbsoluteSum / (compared * 3)).toFixed(4)) : 0, + changedPixels: changed, changedFraction: compared ? Number((changed / compared).toFixed(6)) : 0, + unexpectedTransparentBorderPixels, + note: "Pixel differences are diagnostics only; visually inspect eyes, face, neck and transitions before approving artwork.", + }; +} + +async function pngInfo(path: string, root: string) { + const file = await checkedPath(root, path); + const details = await stat(file); + if (!details.isFile() || details.size < 24 || details.size > MAX_PNG_BYTES) { + throw new AvatarWorkflowError("PIXEL_QA_PNG_INVALID", "QA images must be PNG files under 30 MB."); + } + const handle = await open(file, "r"); + try { + const header = Buffer.alloc(24); + const result = await handle.read(header, 0, 24, 0); + if (result.bytesRead !== 24) throw new AvatarWorkflowError("PIXEL_QA_PNG_INVALID", "Incomplete PNG header."); + return { file, dimensions: parsePngSize(header) }; + } finally { + await handle.close(); + } +} + +async function decodeRgbaPng(file: string, expectedLength: number): Promise { + return new Promise((resolve, reject) => { + let settled = false; + const fail = (message: string) => { + if (settled) return; + settled = true; + child.kill(); + reject(new AvatarWorkflowError("PIXEL_QA_DECODE_FAILED", message)); + }; + const child = spawn("ffmpeg", [ + "-nostdin", "-hide_banner", "-loglevel", "error", + "-i", file, "-frames:v", "1", "-an", + "-pix_fmt", "rgba", "-f", "rawvideo", "pipe:1", + ], { stdio: ["ignore", "pipe", "pipe"], windowsHide: true, signal: AbortSignal.timeout(60_000) }); + const chunks: Buffer[] = []; + let total = 0; + let stderr = ""; + child.stdout.on("data", (chunk: Buffer) => { + total += chunk.length; + if (total > expectedLength) return fail("Decoded image exceeds header dimensions."); + chunks.push(chunk); + }); + child.stderr.on("data", (chunk: Buffer) => { + if (stderr.length < 2000) stderr += chunk.toString("utf8").slice(0, 2000 - stderr.length); + }); + child.on("error", (error) => fail("FFmpeg unavailable or failed: " + error.message)); + child.on("close", (code) => { + if (settled) return; + if (code !== 0 || total !== expectedLength) { + return fail("PNG decode failed or pixel count differs from header: " + stderr.slice(0, 1000)); + } + settled = true; + resolve(Buffer.concat(chunks, total)); + }); + }); +} + +/** PNG files must remain inside the current workspace; FFmpeg never executes via a shell. */ +export async function comparePngFiles( + referencePath: string, candidatePath: string, + region?: PixelRegion, root = process.cwd(), +): Promise { + const [reference, candidate] = await Promise.all([ + pngInfo(referencePath, root), + pngInfo(candidatePath, root), + ]); + const { width, height } = reference.dimensions; + if (candidate.dimensions.width !== width || candidate.dimensions.height !== height) { + throw new AvatarWorkflowError("PIXEL_QA_SIZE_MISMATCH", "Images must have exactly matching dimensions; no implicit resizing."); + } + if (region) validRegion(region, width, height); + const length = width * height * 4; + const [first, second] = await Promise.all([ + decodeRgbaPng(reference.file, length), + decodeRgbaPng(candidate.file, length), + ]); + return compareRgbaPixels(first, second, width, height, region); +} diff --git a/src/cli/index.ts b/src/cli/index.ts index 2cf257e..83da677 100644 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -18,6 +18,11 @@ import { verifyAvatarBuildRecord } from "../character/provenance.js"; import { approveImportedAvatar } from "../character/review-approval.js"; import { FirstVideoSceneError } from "../scene/first-video.js"; import { createFirstVideoFile } from "../scene/create-first-video-file.js"; +import { requestDrawnVariants, statusDrawnVariants, importDrawnVariants } from "../character/drawn-variants.js"; +import { comparePngFiles, parsePixelRegionArgument } from "../character/pixel-qa.js"; +import { comparePngSequence } from "../character/frame-sequence-qa.js"; +import { createBlinkReview } from "../character/blink-review.js"; +import { createCueBinding, verifyCueBinding, CueBindingError } from "../audio/cue-provenance.js"; function print(value: unknown): void { process.stdout.write(JSON.stringify(value, null, 2) + "\n"); @@ -32,12 +37,20 @@ function usage(): never { " puppetctl character import --out --id \n" + " puppetctl character build [--force]\n" + " puppetctl character approve --reviewed\n" + + " puppetctl character variants request [variant-id ...]\n" + + " puppetctl character variants status \n" + + " puppetctl character variants import [--experimental-short-fade] --reviewed-art\n" + + " puppetctl character qa compare [--region x,y,w,h]\n" + + " puppetctl character qa blink [--region x,y,w,h]\n" + + " puppetctl character qa sequence ... [--region x,y,w,h] [--cut-after i,j] [--control-values v1,v2,...]\n" + " puppetctl character landmarks assist \n" + " puppetctl character landmarks propose --provider codex\n" + " puppetctl character landmarks stage --proposal \n" + " puppetctl character landmarks accept --candidate --reviewed\n" + " puppetctl audio align --out [--binary rhubarb] [--force]\n" + - " puppetctl scene new --character [--id name] [--seconds 12] [--fps 24] [--size 1920x1080] [--voice voice.wav --cues mouth.json]\n" + + " puppetctl audio bind --cues --out \n" + + " puppetctl audio verify --cues --binding \n" + + " puppetctl scene new --character [--id name] [--seconds 12] [--fps 24] [--size 1920x1080] [--voice voice.wav --cues mouth.json [--binding mouth.binding.json]]\n" + " puppetctl scene validate \n" + " puppetctl scene plan \n" + " puppetctl scene preview \n" + @@ -53,7 +66,7 @@ async function run(): Promise { if (first === "--help" || first === "help") { process.stdout.write( "PuppetControl (experimental)\n" + - "Available: doctor, character inspect/import/build/approve, landmark assist/propose/stage/accept, audio align, scene new/validate/plan, single-rig scene preview/render.\n" + + "Available: doctor, character inspect/import/build/approve, drawn-variant request/status/import, pixel-QA compare/sequence, landmark assist/propose/stage/accept, audio align, scene new/validate/plan, single-rig scene preview/render.\n" + "Not yet available: reliable one-click landmark inference, animated pose bridges, captions, automatic transcription/TTS.\n", ); return; @@ -78,12 +91,29 @@ async function run(): Promise { ffmpegAvailable: ffmpegFound, rev2dAvailable, codexCliAvailable, - availableFeatures: ["character-inspect", "scene-create", "scene-validate", "transition-policy", "rhubarb-cue-parsing", "rhubarb-align-if-installed", ...(rev2dAvailable ? ["assisted-png-avatar-import", "explicit-visual-approval", "agent-landmark-proposal-staging", ...(codexCliAvailable ? ["optional-codex-vision-proposals"] : []), "avatar-review-sheets", "single-rig-preview", "single-rig-voice-render", "pose-family-hard-cut-render", "experimental-shot-framing"] : [])], + availableFeatures: ["character-inspect", "scene-create", "scene-validate", "transition-policy", "rhubarb-cue-parsing", "rhubarb-align-if-installed", ...(ffmpegFound ? ["png-pixel-qa", "png-sequence-qa"] : []), ...(rev2dAvailable ? ["assisted-png-avatar-import", "explicit-visual-approval", "local-drawn-variants", "agent-landmark-proposal-staging", ...(codexCliAvailable ? ["optional-codex-vision-proposals"] : []), "avatar-review-sheets", "single-rig-preview", "single-rig-voice-render", "pose-family-hard-cut-render", "experimental-shot-framing"] : [])], unavailableFeatures: ["animated-pose-bridges", "automatic-landmark-placement", "automatic-transcription", "text-to-speech", "multi-audio-mixing"], }); return; } + if (first === "audio" && (second === "bind" || second === "verify") && third) { + const flags = process.argv.slice(5); + const destinationFlag = second === "bind" ? "--out" : "--binding"; + if (flags.length !== 4 || flags[0] !== "--cues" || !flags[1] || + flags[2] !== destinationFlag || !flags[3] || + flags[1]!.startsWith("--") || flags[3]!.startsWith("--")) usage(); + if (second === "bind") { + const binding = await createCueBinding(third, flags[1]!, flags[3]!); + print({ valid: true, status: "cue-bytes-bound", binding }); + } else { + const result = await verifyCueBinding(third, flags[1]!, flags[3]!); + print(result); + if (!result.valid) process.exitCode = 1; + } + return; + } + if (first === "audio" && second === "align" && third) { const flags = process.argv.slice(5); const recognized = new Set(["--out", "--binary", "--dialog", "--force"]); @@ -134,6 +164,75 @@ async function run(): Promise { return; } + if (first === "character" && second === "variants" && third) { + const [project, ...flags] = process.argv.slice(5); + if (!project) usage(); + if (third === "status" && flags.length === 0) { + print(await statusDrawnVariants(project)); + return; + } + if (third === "request") { + print(await requestDrawnVariants(project, flags)); + return; + } + if (third === "import" && flags.length >= 2 && flags[flags.length - 1] === "--reviewed-art") { + const requested = flags.slice(0, -1); + const shortFadeIndex = requested.indexOf("--experimental-short-fade"); + if (shortFadeIndex >= 0 && shortFadeIndex !== requested.length - 1) usage(); + const ids = shortFadeIndex >= 0 ? requested.slice(0, -1) : requested; + print(await importDrawnVariants(project, ids, process.cwd(), { experimentalShortFade: shortFadeIndex >= 0 })); + return; + } + usage(); + } + + if (first === "character" && second === "qa" && third === "blink") { + const [project, ...flags] = process.argv.slice(5); + if (!project || (flags.length !== 0 && + (flags.length !== 2 || flags[0] !== "--region" || !flags[1]))) usage(); + print(await createBlinkReview(project, { + ...(flags.length ? { region: parsePixelRegionArgument(flags[1]!) } : {}), + })); + return; + } + + if (first === "character" && second === "qa" && third === "compare") { + const [reference, candidate, ...flags] = process.argv.slice(5); + if (!reference || !candidate || + (flags.length !== 0 && (flags.length !== 2 || flags[0] !== "--region" || !flags[1]))) usage(); + const region = flags.length ? parsePixelRegionArgument(flags[1]!) : undefined; + print({ valid: true, comparison: await comparePngFiles(reference, candidate, region) }); + return; + } + + if (first === "character" && second === "qa" && third === "sequence") { + const args = process.argv.slice(5); + const firstOption = args.findIndex((arg) => arg.startsWith("--")); + const frameFiles = firstOption < 0 ? args : args.slice(0, firstOption); + const flags = firstOption < 0 ? [] : args.slice(firstOption); + if (frameFiles.length < 2 || frameFiles.length > 60) usage(); + let region: ReturnType | undefined; + let cutAfter: number[] = []; + let controlValues: number[] | undefined; + const seen = new Set(); + for (let index = 0; index < flags.length; index += 2) { + const flag = flags[index]; + const value = flags[index + 1]; + if (!flag || !value || seen.has(flag) || !["--region", "--cut-after", "--control-values"].includes(flag)) usage(); + seen.add(flag); + if (flag === "--region") region = parsePixelRegionArgument(value); + else if (flag === "--cut-after") { + if (!/^\d+(,\d+)*$/.test(value)) usage(); + cutAfter = value.split(",").map(Number); + } else { + if (!/^-?(?:\d+(?:\.\d+)?|\.\d+)(,-?(?:\d+(?:\.\d+)?|\.\d+))*$/.test(value)) usage(); + controlValues = value.split(",").map(Number); + } + } + print({ valid: true, report: await comparePngSequence(frameFiles, { ...(region ? { region } : {}), cutAfter, ...(controlValues ? { controlValues } : {}) }) }); + return; + } + if (first === "character" && second === "landmarks" && third) { const [project, ...flags] = process.argv.slice(5); if (!project) usage(); @@ -193,7 +292,7 @@ async function run(): Promise { if (first === "scene" && second === "new" && third) { const flags = process.argv.slice(5); - const allowed = new Set(["--character", "--id", "--seconds", "--fps", "--size", "--voice", "--cues"]); + const allowed = new Set(["--character", "--id", "--seconds", "--fps", "--size", "--voice", "--cues", "--binding"]); const values = new Map(); for (let index = 0; index < flags.length; index += 2) { const flag = flags[index]; @@ -215,6 +314,7 @@ async function run(): Promise { ...(dimensions ? { outputSize: { width: dimensions[0]!, height: dimensions[1]! } } : {}), ...(values.has("--voice") ? { voiceFile: values.get("--voice")! } : {}), ...(values.has("--cues") ? { cuesFile: values.get("--cues")! } : {}), + ...(values.has("--binding") ? { bindingFile: values.get("--binding")! } : {}), })); return; } @@ -298,7 +398,7 @@ async function run(): Promise { run().catch((error: unknown) => { print({ valid: false, - code: error instanceof PreviewUnsupportedError || error instanceof PoseTimelineError || error instanceof AvatarWorkflowError || error instanceof FirstVideoSceneError ? error.code : "CLI_UNEXPECTED_ERROR", + code: error instanceof PreviewUnsupportedError || error instanceof PoseTimelineError || error instanceof AvatarWorkflowError || error instanceof FirstVideoSceneError || error instanceof CueBindingError ? error.code : "CLI_UNEXPECTED_ERROR", message: error instanceof Error ? error.message : String(error), }); process.exitCode = 1; diff --git a/src/render/rev2d.ts b/src/render/rev2d.ts index 687863e..40b4b37 100644 --- a/src/render/rev2d.ts +++ b/src/render/rev2d.ts @@ -6,6 +6,7 @@ import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "nod import type { Character, Scene } from "../scene/schemas.js"; import { compilePerformance, PreviewUnsupportedError } from "./scene-plan.js"; import { parseRhubarbMouthCuesJson } from "../audio/rhubarb.js"; +import { verifyCueBinding } from "../audio/cue-provenance.js"; import { buildFfmpegArgs, type FfmpegAudioInput } from "../audio/ffmpeg.js"; import type { VideoOutput } from "../video/output.js"; import type { SceneEvent } from "../scene/schemas.js"; @@ -117,6 +118,14 @@ async function loadSpeech( const source = await checkedPath(process.cwd(), resolve(directory, entry.path)); const cuesFile = await checkedPath(process.cwd(), resolve(directory, entry.cuesPath)); + if (entry.bindingPath) { + const bindingFile = await checkedPath(process.cwd(), resolve(directory, entry.bindingPath)); + const verified = await verifyCueBinding(source, cuesFile, bindingFile); + if (!verified.valid) { + throw new PreviewUnsupportedError(verified.code, verified.message); + } + } + try { const file = await stat(source); if (!file.isFile() || file.size < 1) throw new Error("Speech file must be a nonempty file"); diff --git a/src/scene/create-first-video-file.ts b/src/scene/create-first-video-file.ts index 18eaf62..b108b11 100644 --- a/src/scene/create-first-video-file.ts +++ b/src/scene/create-first-video-file.ts @@ -9,6 +9,7 @@ import { createFirstVideoScene } from "./first-video.js"; import { checkedPath, AvatarWorkflowError } from "../character/avatar.js"; import { checkAvatarPoseCoverage, verifyAvatarBuildRecord } from "../character/provenance.js"; import { parseRhubarbMouthCuesJson } from "../audio/rhubarb.js"; +import { verifyCueBinding } from "../audio/cue-provenance.js"; export interface CreateFirstVideoFileOptions { outputSceneFile: string; @@ -18,6 +19,8 @@ export interface CreateFirstVideoFileOptions { fps?: number; voiceFile?: string; cuesFile?: string; + /** Optional exact-byte binding created with puppetctl audio bind. */ + bindingFile?: string; outputSize?: { width: number; height: number }; } @@ -76,7 +79,11 @@ export async function createFirstVideoFile( "Use a scene duration of 1–60 seconds and an integer FPS of 1–60."); } const outputDir = dirname(outputFile); - let audio: { path: string; cuesPath: string; startMs: number } | undefined; + let audio: { path: string; cuesPath: string; bindingPath?: string; startMs: number } | undefined; + if (options.bindingFile && (!options.voiceFile || !options.cuesFile)) { + throw new AvatarWorkflowError("FIRST_VIDEO_BINDING_VOICE_REQUIRED", + "A cue binding requires --voice and --cues together."); + } if (Boolean(options.voiceFile) !== Boolean(options.cuesFile)) { throw new AvatarWorkflowError("FIRST_VIDEO_VOICE_CUES_REQUIRED", "Voice requires a Rhubarb cues JSON file; generate it first with puppetctl audio align. No simulated lip sync."); @@ -92,7 +99,15 @@ export async function createFirstVideoFile( throw new AvatarWorkflowError("FIRST_VIDEO_LIP_CUES_INVALID", "Rhubarb cue file is invalid: " + (error instanceof Error ? error.message : String(error))); } - audio = { path: relativeForScene(outputDir, wav), cuesPath: relativeForScene(outputDir, cues), startMs: 0 }; + let bindingPath: string | undefined; + if (options.bindingFile) { + const binding = await checkedPath(project, resolve(project, options.bindingFile)); + const verification = await verifyCueBinding(wav, cues, binding); + if (!verification.valid) throw new AvatarWorkflowError(verification.code, verification.message); + bindingPath = relativeForScene(outputDir, binding); + } + audio = { path: relativeForScene(outputDir, wav), cuesPath: relativeForScene(outputDir, cues), + ...(bindingPath ? { bindingPath } : {}), startMs: 0 }; } const scene = createFirstVideoScene(character, { id: options.id ?? basename(outputFile, extname(outputFile)).replace(/[^A-Za-z0-9_-]/g, "-").replace(/^[^A-Za-z0-9]+/, "").slice(0, 100), @@ -112,7 +127,8 @@ export async function createFirstVideoFile( return { valid: true, status: "editable-scene-created", scene: scene.id, outputFile, character: character.id, rig: modelPath, reusedExistingRig: true, - hasVoice: Boolean(audio), durationMs: scene.durationMs, fps: scene.fps, + hasVoice: Boolean(audio), voiceBoundToCues: Boolean(audio?.bindingPath), + durationMs: scene.durationMs, fps: scene.fps, next: "Edit the scene's supported events, then run scene validate and scene render. The existing approved character and its art are reused unchanged.", }; } diff --git a/src/scene/first-video.ts b/src/scene/first-video.ts index b764687..1dc7ca4 100644 --- a/src/scene/first-video.ts +++ b/src/scene/first-video.ts @@ -16,7 +16,7 @@ export interface FirstVideoOptions { fps: number; resolution: { width: number; height: number }; output?: { width: number; height: number; background: string }; - audio?: { path: string; cuesPath: string; startMs: number }; + audio?: { path: string; cuesPath: string; bindingPath?: string; startMs: number }; /** Supported semantic events in millisecond timestamps. */ events?: SceneEvent[]; } diff --git a/src/scene/load.ts b/src/scene/load.ts index 8914d7e..04a9468 100644 --- a/src/scene/load.ts +++ b/src/scene/load.ts @@ -4,6 +4,7 @@ import { characterSchema, sceneSchema, type Character, type Scene } from "./sche import { validateSceneCapabilities, type Diagnostic, type SceneValidation } from "./validate.js"; import { verifyAvatarBuildRecord, checkAvatarPoseCoverage } from "../character/provenance.js"; import { checkedPath } from "../character/avatar.js"; +import { verifyCueBinding } from "../audio/cue-provenance.js"; async function readJson(path: string): Promise { const text = await readFile(path, "utf8"); @@ -41,6 +42,23 @@ export async function loadAndValidateScene(filePath: string, options: { root?: s const characters = new Map(); const diagnostics: Diagnostic[] = []; + for (const [index, audio] of parsed.data.audio.entries()) { + if (!audio.bindingPath) continue; // Legacy scenes remain supported. + if (!audio.cuesPath) { + diagnostics.push({ severity: "error", code: "CUE_BINDING_MISSING_OR_INVALID", + path: "audio." + index, message: "Bound audio requires Rhubarb cuesPath." }); + continue; + } + const directory = dirname(absolutePath); + const verified = await verifyCueBinding( + resolve(directory, audio.path), resolve(directory, audio.cuesPath), + resolve(directory, audio.bindingPath), + ); + if (!verified.valid) diagnostics.push({ + severity: "error", code: verified.code, + path: "audio." + index + ".bindingPath", message: verified.message, + }); + } for (const actor of parsed.data.actors) { const characterFile = options.root ? await checkedPath(options.root, resolve(dirname(absolutePath), actor.characterPath)) diff --git a/src/scene/schemas.ts b/src/scene/schemas.ts index 6a1d219..cd6ad0e 100644 --- a/src/scene/schemas.ts +++ b/src/scene/schemas.ts @@ -141,6 +141,8 @@ export const sceneSchema = z.strictObject({ startMs: milliseconds, actor: identifier.optional(), cuesPath: z.string().trim().min(1).optional(), + /** Optional SHA-256 sidecar binding the exact voice and Rhubarb cue bytes. */ + bindingPath: z.string().trim().min(1).optional(), })).max(1).default([]), allowCameraCuts: z.boolean().default(false), events: z.array(sceneEventSchema).default([]), diff --git a/tests/blink-review.test.ts b/tests/blink-review.test.ts new file mode 100644 index 0000000..e9bbfb4 --- /dev/null +++ b/tests/blink-review.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, it } from "vitest"; +import { + BLINK_CLOSING_VALUES, BLINK_REVIEW_VALUES, eyeRegionFromAvatarSpec, +} from "../src/character/blink-review.js"; + +describe("non-approving imported-avatar blink review", () => { + it("samples critical sprite transition boundaries without skipping 0.19 or 0.69", () => { + expect(BLINK_CLOSING_VALUES).toContain(0.19); + expect(BLINK_CLOSING_VALUES).toContain(0.69); + expect(BLINK_CLOSING_VALUES).toContain(0.18); + expect(BLINK_CLOSING_VALUES).toContain(0.68); + expect(BLINK_CLOSING_VALUES[0]).toBe(1); + expect(BLINK_CLOSING_VALUES.at(-1)).toBe(0); + }); + + it("mirrors every closing sample into reopening and returns to its starting value", () => { + expect(BLINK_REVIEW_VALUES).toHaveLength(BLINK_CLOSING_VALUES.length * 2 - 1); + expect(BLINK_REVIEW_VALUES.at(-1)).toBe(1); + for (let i = 0; i < BLINK_REVIEW_VALUES.length; i++) { + expect(BLINK_REVIEW_VALUES[i]).toBe(BLINK_REVIEW_VALUES.at(-1 - i)); + } + }); + + it("derives a padded crop containing both iris/eyelid regions", () => { + const spec = { face: { eyes: { + imageLeft: { opening: [165, 206, 202, 202, 205, 225, 160, 230] }, + imageRight: { opening: [305, 210, 346, 208, 349, 227, 302, 232] }, + } } }; + const region = eyeRegionFromAvatarSpec(spec, 512, 640); + expect(region.x).toBeLessThan(160); + expect(region.y).toBeLessThan(200); + expect(region.x + region.width).toBeGreaterThan(349); + expect(region.y + region.height).toBeGreaterThan(232); + }); + + it("refuses to guess when eyes cannot be located", () => { + expect(() => eyeRegionFromAvatarSpec({}, 512, 640)) + .toThrow(/--region/); + expect(() => eyeRegionFromAvatarSpec({ + face: { eyes: { imageLeft: { opening: [0, 1, 2, 3, 4, 5, 6, 7] } } }, + }, 512, 640)).toThrow(/--region/); + }); +}); diff --git a/tests/cue-provenance.test.ts b/tests/cue-provenance.test.ts new file mode 100644 index 0000000..8bc1f8e --- /dev/null +++ b/tests/cue-provenance.test.ts @@ -0,0 +1,153 @@ +import { afterEach, describe, expect, it } from "vitest"; +import { + copyFile, mkdir, mkdtemp, readFile, rm, symlink, unlink, writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { isAbsolute, join, relative } from "node:path"; +import { + CUE_BINDING_FORMAT, + CUE_BINDING_VERSION, + createCueBinding, + verifyCueBinding, +} from "../src/audio/cue-provenance.js"; + +const cleanup: string[] = []; + +afterEach(async () => { + await Promise.all(cleanup.splice(0).map((path) => rm(path, { recursive: true, force: true }))); +}); + +function cuesJson(value = "A") { + return JSON.stringify({ + metadata: { soundFile: "voice.wav", duration: 0.2 }, + mouthCues: [{ start: 0, end: 0.2, value }], + }) + "\n"; +} + +async function fixture() { + const cache = join(process.cwd(), ".cache"); + await mkdir(cache, { recursive: true }); + const dir = await mkdtemp(join(cache, "cue-provenance-test-")); + cleanup.push(dir); + const audio = join(dir, "voice.wav"); + const cues = join(dir, "voice.cues.json"); + const sidecar = join(dir, "voice.cues.binding.json"); + await writeFile(audio, Buffer.from("RIFF-test-audio-bytes-v1")); + await writeFile(cues, cuesJson()); + return { dir, audio, cues, sidecar }; +} + +function canonical(path: string) { + return relative(process.cwd(), path).replaceAll("\\", "/"); +} + +describe("Rhubarb cue provenance sidecar", () => { + it("creates deterministic v1 metadata bound to exact bytes and canonical workspace paths", async () => { + const { audio, cues, sidecar } = await fixture(); + + const binding = await createCueBinding(audio, cues, sidecar); + expect(binding).toMatchObject({ + format: CUE_BINDING_FORMAT, + version: CUE_BINDING_VERSION, + hashAlgorithm: "sha256", + sidecarPath: canonical(sidecar), + audio: { path: canonical(audio) }, + cues: { path: canonical(cues) }, + }); + expect(binding.audio.sha256).toMatch(/^[a-f0-9]{64}$/); + expect(binding.cues.sha256).toMatch(/^[a-f0-9]{64}$/); + expect(isAbsolute(binding.audio.path)).toBe(false); + expect(isAbsolute(binding.cues.path)).toBe(false); + expect(isAbsolute(binding.sidecarPath)).toBe(false); + + const disk = JSON.parse(await readFile(sidecar, "utf8")) as unknown; + expect(disk).toEqual(binding); + expect(await verifyCueBinding(audio, cues, sidecar)).toEqual({ valid: true, binding }); + }); + + it("refuses to bind malformed or empty Rhubarb cue data", async () => { + const malformed = await fixture(); + await writeFile(malformed.cues, JSON.stringify({ metadata: {}, mouthCues: [] })); + await expect(createCueBinding(malformed.audio, malformed.cues, malformed.sidecar)) + .rejects.toThrow("at least one mouth cue"); + expect(await verifyCueBinding(malformed.audio, malformed.cues, malformed.sidecar)) + .toMatchObject({ valid: false, code: "CUE_BINDING_MISSING_OR_INVALID" }); + }); + + it("fails closed if either audio or cue bytes change", async () => { + const first = await fixture(); + await createCueBinding(first.audio, first.cues, first.sidecar); + await writeFile(first.audio, Buffer.from("RIFF-tampered-audio")); + expect(await verifyCueBinding(first.audio, first.cues, first.sidecar)) + .toMatchObject({ valid: false, code: "CUE_BINDING_STALE" }); + + const second = await fixture(); + await createCueBinding(second.audio, second.cues, second.sidecar); + await writeFile(second.cues, cuesJson("B")); + expect(await verifyCueBinding(second.audio, second.cues, second.sidecar)) + .toMatchObject({ valid: false, code: "CUE_BINDING_STALE" }); + }); + + it("fails closed on missing, malformed, or unsupported sidecars", async () => { + const { audio, cues, sidecar } = await fixture(); + expect(await verifyCueBinding(audio, cues, sidecar)) + .toMatchObject({ valid: false, code: "CUE_BINDING_MISSING_OR_INVALID" }); + + await writeFile(sidecar, "{"); + expect(await verifyCueBinding(audio, cues, sidecar)) + .toMatchObject({ valid: false, code: "CUE_BINDING_MISSING_OR_INVALID" }); + + await rm(sidecar); + const binding = await createCueBinding(audio, cues, sidecar); + await writeFile(sidecar, JSON.stringify({ ...binding, version: 2 })); + expect(await verifyCueBinding(audio, cues, sidecar)) + .toMatchObject({ valid: false, code: "CUE_BINDING_MISSING_OR_INVALID" }); + }); + + it("binds the sidecar to its canonical location and refuses silent rebinding", async () => { + const { dir, audio, cues, sidecar } = await fixture(); + const binding = await createCueBinding(audio, cues, sidecar); + + await expect(createCueBinding(audio, cues, sidecar)) + .rejects.toMatchObject({ code: "CUE_BINDING_EXISTS" }); + expect(JSON.parse(await readFile(sidecar, "utf8"))).toEqual(binding); + + const copied = join(dir, "copied.binding.json"); + await copyFile(sidecar, copied); + expect(await verifyCueBinding(audio, cues, copied)) + .toMatchObject({ valid: false, code: "CUE_BINDING_STALE" }); + }); + + it("canonicalizes safe in-workspace symlink aliases", async () => { + const { dir, cues, sidecar } = await fixture(); + const targetDir = join(dir, "real-audio"); + const aliasDir = join(dir, "audio-alias"); + await mkdir(targetDir); + const audio = join(targetDir, "voice.wav"); + await writeFile(audio, Buffer.from("RIFF-symlinked-audio")); + await symlink(targetDir, aliasDir, process.platform === "win32" ? "junction" : "dir"); + + const throughAlias = join(aliasDir, "voice.wav"); + const binding = await createCueBinding(throughAlias, cues, sidecar); + expect(binding.audio.path).toBe(canonical(audio)); + expect(await verifyCueBinding(throughAlias, cues, sidecar)).toEqual({ valid: true, binding }); + }); + + it("rejects symlink escapes outside the workspace for inputs and sidecars", async () => { + const { dir, audio, cues, sidecar } = await fixture(); + const outside = await mkdtemp(join(tmpdir(), "puppetcontrol-cue-outside-")); + cleanup.push(outside); + await writeFile(join(outside, "outside.wav"), Buffer.from("RIFF-outside")); + const escape = join(dir, "escape"); + await symlink(outside, escape, process.platform === "win32" ? "junction" : "dir"); + + await expect(createCueBinding(join(escape, "outside.wav"), cues, sidecar)) + .rejects.toMatchObject({ code: "PATH_OUTSIDE_WORKSPACE" }); + + const outsideSidecar = join(escape, "binding.json"); + await expect(createCueBinding(audio, cues, outsideSidecar)) + .rejects.toMatchObject({ code: "PATH_OUTSIDE_WORKSPACE" }); + expect(await verifyCueBinding(audio, cues, outsideSidecar)) + .toMatchObject({ valid: false, code: "CUE_BINDING_MISSING_OR_INVALID" }); + }); +}); diff --git a/tests/drawn-variants.test.ts b/tests/drawn-variants.test.ts new file mode 100644 index 0000000..641f600 --- /dev/null +++ b/tests/drawn-variants.test.ts @@ -0,0 +1,103 @@ +import { afterEach, describe, expect, it } from "vitest"; +import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createImportedCharacter, runRev2D } from "../src/character/avatar.js"; +import { createAvatarBuildRecord, writeAvatarBuildRecord } from "../src/character/provenance.js"; +import { + DRAWN_VARIANT_IDS, importDrawnVariants, requestDrawnVariants, validateDrawnVariantIds, +} from "../src/character/drawn-variants.js"; + +const projects: string[] = []; +afterEach(async () => { + await Promise.all(projects.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); +}); + +function pngHeader(width: number, height: number): Buffer { + const bytes = Buffer.alloc(24); + Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]).copy(bytes); + bytes.write("IHDR", 12, "ascii"); + bytes.writeUInt32BE(width, 16); + bytes.writeUInt32BE(height, 20); + return bytes; +} + +async function fixture() { + const dir = await mkdtemp(join(tmpdir(), "puppetcontrol-variants-")); + projects.push(dir); + await mkdir(join(dir, "textures")); + await mkdir(join(dir, "variants")); + await writeFile(join(dir, "source.png"), pngHeader(512, 640)); + await writeFile(join(dir, "avatar.json"), '{"format":"rev2d-avatar","name":"test","image":{"file":"source.png"}}'); + await writeFile(join(dir, "textures", "base.png"), "texture"); + await writeFile(join(dir, "test.r2d.json"), JSON.stringify({ + textures: [{ id: "base", path: "textures/base.png", width: 512, height: 640 }], + })); + const parameters = ["AngleX", "AngleY", "EyeBallX", "EyeBallY", "MouthOpen", "MouthForm"] + .map((id) => ({ id, min: -1, max: 1, default: 0 })); + const manifest = createImportedCharacter("test", { parameters }); + manifest.poseFamilies[0]!.approved = true; + await writeFile(join(dir, "character.json"), JSON.stringify(manifest)); + await writeAvatarBuildRecord(dir, await createAvatarBuildRecord(dir, "test.r2d.json")); + return dir; +} + +describe("local drawn avatar variants", () => { + it("defaults to only the two recommended variants", () => { + expect(validateDrawnVariantIds([])).toEqual(["eyes_closed", "mouth_a"]); + expect(DRAWN_VARIANT_IDS).toContain("mouth_o"); + expect(validateDrawnVariantIds(["eyes_closed", "eyes_half"])).toEqual(["eyes_closed", "eyes_half"]); + }); + + it("requires explicit IDs for imports and rejects unsupported, duplicate, and option-like IDs", () => { + expect(() => validateDrawnVariantIds([], true)).toThrowError(/exact reviewed artwork/); + for (const ids of [["eyes_closed", "eyes_closed"], ["--all"], ["../../art"], ["mouth_x"]]) { + expect(() => validateDrawnVariantIds(ids)).toThrowError(/variant ID/); + } + }); + + it("refuses stale imported avatar assets instead of laundering changed model bytes", async () => { + const dir = await fixture(); + const modelFile = join(dir, "test.r2d.json"); + const model = JSON.parse(await readFile(modelFile, "utf8")); + model.meta = { width: 512, height: 640 }; + await writeFile(modelFile, JSON.stringify(model)); + await expect(requestDrawnVariants(dir, ["eyes_closed"], dir)) + .rejects.toMatchObject({ code: "AVATAR_ASSETS_STALE" }); + const json = JSON.parse(await readFile(join(dir, "character.json"), "utf8")); + expect(json.poseFamilies[0].approved).toBe(true); + }); + + it("rejects mismatched illustration size BEFORE modifying the approved manifest", async () => { + const dir = await fixture(); + await writeFile(join(dir, "variants", "eyes_closed.png"), pngHeader(512, 600)); + await expect(importDrawnVariants(dir, ["eyes_closed"], dir)) + .rejects.toMatchObject({ code: "VARIANT_SIZE_MISMATCH" }); + const json = JSON.parse(await readFile(join(dir, "character.json"), "utf8")); + expect(json.poseFamilies[0].approved).toBe(true); + }); + + it("rejects missing selected drawings without revoking the original rig", async () => { + const dir = await fixture(); + await expect(importDrawnVariants(dir, ["mouth_a"], dir)).rejects.toThrow(); + const json = JSON.parse(await readFile(join(dir, "character.json"), "utf8")); + expect(json.poseFamilies[0].approved).toBe(true); + }); + + it("surfaces Rev2D drawing rejection codes and reasons even when diagnostics is empty", async () => { + const dir = await fixture(); + await mkdir(join(dir, "bin")); + await writeFile(join(dir, "bin", "rev2d.js"), [ + "process.stdout.write(JSON.stringify({", + " ok: false, diagnostics: [],", + " checks: [{ id: 'eyes_closed', ok: false, code: 'unchanged', reason: 'No pixels changed' }]", + "}));", + "process.exit(1);", + ].join("\n")); + await expect(runRev2D(dir, ["avatar", "variants", "import", dir])) + .rejects.toMatchObject({ + code: "REV2D_AVATAR_FAILED", + message: expect.stringContaining("unchanged"), + }); + }); +}); diff --git a/tests/eye-short-fade.test.ts b/tests/eye-short-fade.test.ts new file mode 100644 index 0000000..a7619b9 --- /dev/null +++ b/tests/eye-short-fade.test.ts @@ -0,0 +1,121 @@ +import { describe, expect, it } from "vitest"; +import { shortFadeDrawnEyes } from "../src/character/eye-short-fade.js"; + +type B = { id: string; target: string; params: string[]; keys: { at: number[]; value: number }[] }; +const values = (entries: Array<[number, number]>) => entries.map(([at, value]) => ({ at: [at], value })); +const sprite = (name: string, param: string, initial: number): B => ({ + id: name + "Show", target: "part:" + name + ".opacity", params: [param], + keys: values([[0, initial], [1, 0]]), +}); +function fixture() { + const bindings: B[] = [ + sprite("sprite_eyes_closed_imageLeft", "EyeROpen", 1), + sprite("sprite_eyes_closed_imageRight", "EyeLOpen", 1), + sprite("sprite_eyes_half_imageLeft", "EyeROpen", 1), + sprite("sprite_eyes_half_imageRight", "EyeLOpen", 1), + ]; + for (const [eye, param] of [["eyeR", "EyeROpen"], ["eyeL", "EyeLOpen"]]) { + for (const kind of ["white", "ball", "lower", "upper", "crease"]) { + const name = eye + "_" + kind; + bindings.push({ + id: name + "UnderSprite", target: "part:" + name + ".opacity", + params: [param], keys: values([[0, 0], [0.7, 1], [1, 1]]), + }); + } + } + bindings.push({ id: "mouthShape", target: "part:mouth.opacity", + params: ["MouthOpen"], keys: values([[0, 1], [1, 1]]) }); + return { meta: { name: "experimental", width: 512, height: 640 }, bindings }; +} + +function sample(keys: { at: number[]; value: number }[], x: number) { + if (x <= keys[0]!.at[0]!) return keys[0]!.value; + for (let i = 1; i < keys.length; i++) { + const a = keys[i - 1]!, b = keys[i]!; + if (x <= b.at[0]!) { + const fraction = (x - a.at[0]!) / (b.at[0]! - a.at[0]!); + return a.value + (b.value - a.value) * fraction; + } + } + return keys[keys.length - 1]!.value; +} + +describe("experimental short fade for reviewed eye sprites", () => { + it("returns a new model with only expected eye opacity bindings changed", () => { + const src = fixture(); + const before = structuredClone(src); + const result = shortFadeDrawnEyes(src); + const bindings = result.model.bindings as B[]; + expect(src).toEqual(before); + expect(result.model).not.toBe(src); + expect(result.changedBindings).toHaveLength(14); + expect(bindings.find((b) => b.id === "mouthShape")).toEqual(before.bindings.at(-1)); + for (const side of ["imageLeft", "imageRight"]) { + const closed = bindings.find((b) => b.id === "sprite_eyes_closed_" + side + "Show")!; + const half = bindings.find((b) => b.id === "sprite_eyes_half_" + side + "Show")!; + expect(sample(closed.keys, 0.167)).toBe(1); + expect(sample(half.keys, 0.167)).toBe(0); + expect(sample(half.keys, 0.5)).toBe(1); + expect(sample(half.keys, 0.667)).toBe(1); + expect(sample(half.keys, 0.833)).toBe(0); + expect(sample(closed.keys, 0.19)).toBeCloseTo(0.5, 7); + expect(sample(half.keys, 0.19)).toBeCloseTo(0.5, 7); + // It is a short fade, NOT a claim that transitions can never ghost. + } + expect(result.profile.closedToHalf).toEqual([0.18, 0.2]); + }); + + it("keeps both eyes fully hidden in drawn poses, and restores mesh outside the sprites", () => { + const model = shortFadeDrawnEyes(fixture()).model; + const mesh = (model.bindings as B[]).filter((b) => b.id.endsWith("UnderSprite")); + expect(mesh).toHaveLength(10); + for (const b of mesh) { + expect(sample(b.keys, 0.5)).toBe(0); + expect(sample(b.keys, 0.667)).toBe(0); + expect(sample(b.keys, 0.7)).toBe(1); + expect(sample(b.keys, 1)).toBe(1); + } + }); + + it("preserves upstream multi-parameter warp bindings even though their values are not scalars", () => { + const original = fixture(); + const warp = { + target: "warp:avatarWarp.offsets", params: ["AngleX", "AngleY"], + keys: [{ at: [-30, -30], value: { "264": [-0.01, 0.01] } }], + }; + const model = { ...original, bindings: [...original.bindings, warp] }; + const before = structuredClone(model); + const transformed = shortFadeDrawnEyes(model).model as { bindings: unknown[] }; + expect(model).toEqual(before); + expect(transformed.bindings.at(-1)).toEqual(warp); + }); + + it("refuses a missing half-eye sprite or incomplete mesh-hide rig", () => { + const a = fixture(); + a.bindings = a.bindings.filter((b) => b.id !== "sprite_eyes_half_imageRightShow"); + expect(() => shortFadeDrawnEyes(a)).toThrow(/half.*closed variants/); + const b = fixture(); + b.bindings = b.bindings.filter((entry) => !entry.id.startsWith("eyeR_") || entry.id.includes("white")); + expect(() => shortFadeDrawnEyes(b)).toThrow(/incomplete eye-mesh/); + }); + + it("refuses unknown eye smile layering, duplicate bindings or incorrect eye parameter mapping", () => { + const smile = fixture(); + smile.bindings.push(sprite("sprite_eyes_smile_imageLeft", "EyeRSmile", 1)); + expect(() => shortFadeDrawnEyes(smile)).toThrow(/eyes_smile/); + const duplicate = fixture(); + duplicate.bindings.push(duplicate.bindings[0]!); + expect(() => shortFadeDrawnEyes(duplicate)).toThrow(/duplicate/); + const mismatch = fixture(); + mismatch.bindings[0]!.params = ["EyeLOpen"]; + expect(() => shortFadeDrawnEyes(mismatch)).toThrow(/incompatible/); + }); + + it("refuses unsupported model layouts and nonstandard mesh-opacity values", () => { + expect(() => shortFadeDrawnEyes(null)).toThrow(/invalid model/); + expect(() => shortFadeDrawnEyes({ bindings: [] })).toThrow(/missing or incompatible/); + const invalid = fixture(); + invalid.bindings[4]!.keys[0]!.value = 0.1; + expect(() => shortFadeDrawnEyes(invalid)).toThrow(/nonstandard/); + }); +}); diff --git a/tests/first-video-file.test.ts b/tests/first-video-file.test.ts index edbcb0f..32b8f1f 100644 --- a/tests/first-video-file.test.ts +++ b/tests/first-video-file.test.ts @@ -3,6 +3,7 @@ import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import { join, resolve } from "node:path"; import { createFirstVideoFile } from "../src/scene/create-first-video-file.js"; import { loadAndValidateScene } from "../src/scene/load.js"; +import { createCueBinding } from "../src/audio/cue-provenance.js"; let root = ""; let working = ""; @@ -56,6 +57,32 @@ describe("first-video file creation", () => { expect(first.actors[0].characterPath).toBe(second.actors[0].characterPath); }); + it("binds voice and Rhubarb cues to immutable bytes and refuses stale audio before rendering", async () => { + const output = join(working, "bound-scene.json"); + const wav = join(working, "voice.wav"); + const cues = join(working, "voice.cues.json"); + const sidecar = join(working, "voice.binding.json"); + await writeFile(wav, Buffer.from("RIFF-test-voice-content")); + await writeFile(cues, JSON.stringify({ + metadata: { soundFile: "voice.wav", duration: 0.2 }, + mouthCues: [{ start: 0, end: 0.2, value: "A" }], + })); + await createCueBinding(wav, cues, sidecar); + const created = await createFirstVideoFile({ + outputSceneFile: output, characterFile: hana, voiceFile: wav, cuesFile: cues, + bindingFile: sidecar, seconds: 3, + }, root); + expect(created.voiceBoundToCues).toBe(true); + const first = await loadAndValidateScene(output); + expect(first.valid).toBe(true); + await writeFile(wav, Buffer.from("RIFF-tampered-voice")); + const next = await loadAndValidateScene(output); + expect(next.valid).toBe(false); + if (!next.valid) { + expect(next.diagnostics.map((diagnostic) => diagnostic.code)).toContain("CUE_BINDING_STALE"); + } + }); + it("derives distinct default scene IDs from distinct filenames", async () => { const first = join(working, "first-clip.json"); const second = join(working, "second-clip.json"); diff --git a/tests/frame-sequence-qa.test.ts b/tests/frame-sequence-qa.test.ts new file mode 100644 index 0000000..1998153 --- /dev/null +++ b/tests/frame-sequence-qa.test.ts @@ -0,0 +1,117 @@ +import { describe, expect, it } from "vitest"; +import { analyzeSequenceComparisons, comparePngSequence, MAX_SEQUENCE_FRAMES } from "../src/character/frame-sequence-qa.js"; +import type { PixelComparison } from "../src/character/pixel-qa.js"; + +const region = { x: 0, y: 0, width: 256, height: 256 }; +function metric(error: number, changedFraction = error / 255): PixelComparison { + return { + width: 256, height: 256, region, comparedPixels: 65_536, + ignoredTransparentPixels: 0, rgbMeanAbsoluteError: error, + changedPixels: Math.round(65_536 * changedFraction), changedFraction, + unexpectedTransparentBorderPixels: 0, note: "manual review only", + }; +} + +describe("time-ordered visual motion QA", () => { + it("reports a single anomalous pair without claiming artistic failure", () => { + const report = analyzeSequenceComparisons( + [metric(3), metric(3), metric(35), metric(3), metric(3)], metric(0), + ); + expect(report.frameCount).toBe(6); + expect(report.medianPairError).toBe(3); + expect(report.possibleAbruptChanges).toEqual([2]); + expect(report.pairs[2]).toMatchObject({ + from: 2, to: 3, abruptChangeCandidate: true, intentionalCut: false, + }); + expect(report.firstToLast.rgbMeanAbsoluteError).toBe(0); + expect(report.possibleMotionStalls).toEqual([]); + expect(report.controlValuesProvided).toBe(false); + expect(report.note).toMatch(/triage/i); + }); + + it("does not flag an explicitly authored editorial cut or poison baseline with its high score", () => { + const report = analyzeSequenceComparisons( + [metric(2), metric(2), metric(175), metric(2), metric(2)], metric(170), { cutAfter: [2] }, + ); + expect(report.medianPairError).toBe(2); + expect(report.possibleAbruptChanges).toEqual([]); + expect(report.pairs[2]).toMatchObject({ + abruptChangeCandidate: false, intentionalCut: true, + }); + }); + + it("refuses to classify a two/three-frame sample as a proven jump", () => { + const small = analyzeSequenceComparisons([metric(210)], metric(210)); + expect(small.possibleAbruptChanges).toEqual([]); + const next = analyzeSequenceComparisons([metric(210), metric(1)], metric(0)); + expect(next.possibleAbruptChanges).toEqual([]); + }); + + it("tolerates smooth regular steps and zero movement", () => { + const regular = analyzeSequenceComparisons( + [metric(11), metric(10), metric(12), metric(11), metric(10)], metric(1), + ); + expect(regular.possibleAbruptChanges).toEqual([]); + const frozen = analyzeSequenceComparisons( + [metric(0), metric(0), metric(0)], metric(0), + ); + expect(frozen.possibleAbruptChanges).toEqual([]); + expect(frozen.firstToLast.changedFraction).toBe(0); + }); + + it("flags unchanged frames when an explicitly sampled control moved", () => { + const report = analyzeSequenceComparisons( + [metric(5), metric(0), metric(5), metric(0)], metric(0), + { controlValues: [1, 0.8, 0.6, 0.4, 0.2] }, + ); + expect(report.controlValuesProvided).toBe(true); + expect(report.possibleMotionStalls).toEqual([1, 3]); + expect(report.pairs[1]).toMatchObject({ + from: 1, to: 2, motionStallCandidate: true, controlDelta: 0.2, + }); + const noChangeExpected = analyzeSequenceComparisons( + [metric(0), metric(0), metric(0)], metric(0), + { controlValues: [1, 1, 1, 1] }, + ); + expect(noChangeExpected.possibleMotionStalls).toEqual([]); + }); + + it("refuses short, non-finite or unrelated control arrays", () => { + expect(() => analyzeSequenceComparisons([metric(0), metric(0)], metric(0), + { controlValues: [1, 0.8] })).toThrow(/one finite value/); + expect(() => analyzeSequenceComparisons([metric(0), metric(0)], metric(0), + { controlValues: [1, NaN, 0] })).toThrow(/one finite value/); + expect(() => analyzeSequenceComparisons([metric(0), metric(0)], metric(0), + { controlValues: [1, Infinity, 0] })).toThrow(/one finite value/); + }); + + it("rejects malformed and duplicate cut indexes, excessive sequences and inconsistent regions", () => { + expect(MAX_SEQUENCE_FRAMES).toBe(60); + for (const cuts of [[2, 2], [-1], [4], [1.5], [NaN]]) { + expect(() => analyzeSequenceComparisons([metric(0),metric(0),metric(0)], metric(0), { cutAfter: cuts })) + .toThrow(/cut/i); + } + expect(() => analyzeSequenceComparisons([], metric(0))).toThrow(/2–60/); + expect(() => analyzeSequenceComparisons(Array.from({ length: 60 }, () => metric(0)), metric(0))) + .toThrow(/2–60/); + const other = { ...metric(4), region: { x: 1, y: 0, width: 255, height: 256 } }; + expect(() => analyzeSequenceComparisons([metric(1), other, metric(1)], metric(0))) + .toThrow(/same exact/); + }); + + it("refuses out-of-bounds and nonfinite comparison metrics", () => { + for (const bad of [NaN, Infinity, -1, 256]) { + expect(() => analyzeSequenceComparisons([metric(0), metric(bad), metric(0)], metric(0))) + .toThrow(/bounded numeric metrics/); + } + expect(() => analyzeSequenceComparisons([metric(0, 1.1)], metric(0))) + .toThrow(/bounded numeric metrics/); + }); + + it("requires explicitly ordered PNG filenames before starting external decoding", async () => { + await expect(comparePngSequence(["a.jpg", "b.png"])) + .rejects.toMatchObject({ code: "SEQUENCE_QA_PNG_REQUIRED" }); + await expect(comparePngSequence(["a.png"])) + .rejects.toMatchObject({ code: "SEQUENCE_QA_LENGTH_INVALID" }); + }); +}); diff --git a/tests/pixel-qa.test.ts b/tests/pixel-qa.test.ts new file mode 100644 index 0000000..60d0828 --- /dev/null +++ b/tests/pixel-qa.test.ts @@ -0,0 +1,79 @@ +import { describe, expect, it } from "vitest"; +import { compareRgbaPixels, parsePixelRegionArgument } from "../src/character/pixel-qa.js"; + +function solid(width: number, height: number, rgba: [number, number, number, number]): Uint8Array { + const bytes = new Uint8Array(width * height * 4); + for (let i = 0; i < width * height; i++) bytes.set(rgba, i * 4); + return bytes; +} + +describe("pixel-level image comparison is deterministic triage, not automatic approval", () => { + it("compares identical pixels without invented motion or border gaps", () => { + const reference = solid(4, 3, [20, 40, 60, 255]); + const result = compareRgbaPixels(reference, reference, 4, 3); + expect(result).toMatchObject({ + comparedPixels: 12, ignoredTransparentPixels: 0, + rgbMeanAbsoluteError: 0, changedPixels: 0, changedFraction: 0, + unexpectedTransparentBorderPixels: 0, + }); + expect(result.note).toMatch(/visually inspect/i); + }); + + it("measures region-specific RGB change without counting unaffected pixels", () => { + const reference = solid(3, 3, [10, 20, 30, 255]); + const changed = reference.slice(); + const center = (1 * 3 + 1) * 4; + changed[center] = 100; + const full = compareRgbaPixels(reference, changed, 3, 3); + expect(full.changedPixels).toBe(1); + expect(full.rgbMeanAbsoluteError).toBeCloseTo(90 / (9 * 3), 3); + const crop = compareRgbaPixels(reference, changed, 3, 3, + { x: 1, y: 1, width: 1, height: 1 }); + expect(crop.changedFraction).toBe(1); + expect(crop.rgbMeanAbsoluteError).toBe(30); + }); + + it("ignores fully transparent RGB noise and finds new opaque-border holes", () => { + const reference = solid(3, 3, [100, 120, 140, 255]); + const candidate = reference.slice(); + // Reference-opaque top-left corner becomes transparent. + candidate[3] = 0; + // Fully transparent center has arbitrary irrelevant RGB values. + const emptyReference = solid(2, 2, [12, 0, 0, 0]); + const emptyCandidate = solid(2, 2, [250, 180, 160, 0]); + const transparent = compareRgbaPixels(emptyReference, emptyCandidate, 2, 2); + expect(transparent.comparedPixels).toBe(0); + expect(transparent.ignoredTransparentPixels).toBe(4); + expect(transparent.changedPixels).toBe(0); + expect(compareRgbaPixels(reference, candidate, 3, 3).unexpectedTransparentBorderPixels).toBe(1); + }); + + it("does not double-count corners or single-row/single-column borders", () => { + const ref = solid(1, 3, [3, 4, 5, 255]); + const result = compareRgbaPixels(ref, solid(1, 3, [3, 4, 5, 0]), 1, 3); + expect(result.unexpectedTransparentBorderPixels).toBe(3); + const flat = compareRgbaPixels(solid(3, 1, [0, 0, 0, 255]), + solid(3, 1, [0, 0, 0, 0]), 3, 1); + expect(flat.unexpectedTransparentBorderPixels).toBe(3); + }); + + it("parses strict CLI region syntax and rejects malformed values", () => { + expect(parsePixelRegionArgument("175,220,170,75")) + .toEqual({ x: 175, y: 220, width: 170, height: 75 }); + for (const bad of ["0,0,0,1", "-1,0,1,1", "2,3,4", "1.5,0,1,1", "1,2,3,4junk", "1,2,3,9999999999999999999"]) { + expect(() => parsePixelRegionArgument(bad)).toThrowError(/region/i); + } + }); + + it("rejects wrong pixel lengths, unsafe sizes and ROIs", () => { + const img = solid(2, 2, [0, 0, 0, 255]); + expect(() => compareRgbaPixels(img, img.slice(0, 8), 2, 2)) + .toThrowError(/matching RGBA/); + expect(() => compareRgbaPixels(img, img, 2, 2, { x: 1, y: 1, width: 2, height: 2 })) + .toThrowError(/rectangle/); + expect(() => compareRgbaPixels(img, img, 2, 2, { x: 0, y: 0, width: 0, height: 1 })) + .toThrowError(/rectangle/); + expect(() => compareRgbaPixels(img, img, Infinity, 2)) + .toThrowError(/bounded dimensions/); + }); +});