Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .agents/skills/puppetcontrol/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Read repo-root AGENTS.md, docs/TECHNICAL_CONCEPT.md, and docs/CHARACTER_IMPORT.m
9. Within a compatible shared rig, semantic pose events may interpolate through validated parameter presets (the neutral-to-pointing demo uses shoulder and elbow controls). Distinct pose models require explicit scene camera cuts; an authored bridge is NOT implemented. Never silently crossfade meshes or call a body cut a smooth physical transition.
10. For speech, prepare WAV, install Rhubarb separately, and use node dist/cli/index.js audio align <voice.wav> --out <mouth.json> [--binary <rhubarb>]. Reference audio.path, audio.startMs and audio.cuesPath from the scene; review the actual mouth/audio timing.
11. Do not commit private character illustrations, full generated models, user voice recordings or secrets without checking redistribution rights. Demo:avatar generates pinned MIT-licensed sample artwork into ignored .cache.
12. State what actually rendered and which controls remain unsupported. Do not claim a flat PNG is automatically converted into production-quality animation without landmark review.
12. For animation QA, compare real ordered PNGs with `node dist/cli/index.js character qa sequence <f0.png> <f1.png> ... [--region x,y,w,h] [--control-values v0,v1,...] [--cut-after i,j]`. Supply sampled control values when investigating invisible or stalled motion; label known cuts. Inspect the PNGs/video and report possible ghosting even if the motion statistics look normal. The QA report is not an approval.
13. State what actually rendered and which controls remain unsupported. Do not claim a flat PNG is automatically converted into production-quality animation without landmark review.

Keep edits small and testable. Run npm run verify and inspect visual evidence for any animation/asset changes.
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ The long-term goal: **generate artwork when necessary, animate and reuse it inde
- Assisted single-PNG character onboarding through Rev2D landmarks, local rig build, review sheets, unapproved-by-default manifest, and starter scene.
- Agent-guided landmark proposals: source-hash-bound task, optional **tested** Codex image-based suggestion, isolated geometry validation, visual overlay and explicit review/acceptance that revokes old rig approval.
- Imported avatar asset provenance: source/spec/model/texture fingerprints checked by scene validation and render; untracked alternate imported pose models are refused. Force rebuild revokes approval before modifying art. Suspicious landmarks generate nonblocking geometry warnings.
- Local PNG pixel and ordered-frame QA (full frame or explicit region) reports RGB activity, transparent borders, candidate motion jumps and changed-control stalls; metrics never auto-approve a rig.
- Agent-readable CLI output, tests and CI.

**Not implemented yet:** fully automatic landmark placement or reliable one-click rigging, AI-generated arbitrary new pose rigs, convincing smooth bridges between unrelated meshes (especially crossed arms), automatic transcription/TTS, captions, B-roll and multiple actors. The arm-raise gesture is genuinely animated, but its artwork is still prototype-quality. Other pose families may require edited hard cuts. The pose manifest is capability metadata; visual poses become usable only after their rig files are generated, validated and inspected.
Expand All @@ -42,6 +43,9 @@ npm run demo:directed # Apply edited camera framing and gradual punch-in
npm run demo:gesture # Generate two-joint gesture rig and export edited 1080p demo
npm run demo:gesture-cycle # Render continuous raise/lower cycles without cuts
npm run demo:avatar # Import a freshly rendered flat PNG and create an unapproved rig + review sheets
# Compare captured PNG frames:
# node dist/cli/index.js character qa compare outputs/rest.png outputs/closed.png --region 175,220,170,75
# node dist/cli/index.js character qa sequence outputs/f00.png outputs/f01.png outputs/f02.png --control-values 1,0.5,0
~~~

The sample uses Rev2D's own **Hana** rig from the pinned MIT-licensed upstream checkout; generated character rig JSON is not checked into this repository. The derived example video is separately attributed and licensed under the original Rev2D terms.
Expand Down Expand Up @@ -94,6 +98,7 @@ Our native animation backend for the spike is **Rev2D**, pinned by the setup hel
- [Agent-guided landmark review and acceptance](docs/LANDMARK_AGENT_WORKFLOW.md)
- [Optional Codex vision inference: real-image results and limitations](docs/CODEX_VISION_LANDMARKS.md)
- [Deep quality and reliability audit with observed pixel measurements](docs/QUALITY_AUDIT.md)
- [Pixel-level visual QA (non-approval diagnostics)](docs/PIXEL_QA.md)
- [Architecture decision log](docs/DECISIONS.md)

Audio/lip-sync works with precomputed Rhubarb cues and a locally installed executable. The smooth pointing demo reuses the same face and deforms one rig continuously; crossed arms still need a more convincing bridge or intentional cut. Next: polish eyelid and mouth deformation by reusing Rev2D's reviewed drawn variants, expand automated image-quality probes and evaluate diverse source artwork. Do not confuse data validation with visual quality.
Expand Down
4 changes: 3 additions & 1 deletion docs/IMPLEMENTATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,9 @@
- [x] Add conservative geometry triage for low torso pivot/fade, missing/outside-eye iris and shearing head pivot; return nonblocking warnings.
- [x] Revoke existing approval before any --force build mutates Rev2D outputs, and fail closed on stale source/spec/model/texture fingerprints.
- [x] Eliminate reproduced black edge artifacts caused by alpha-zero warped pixels in H.264: composite frames onto scene background, with a 60-frame before/after comparison.
- [ ] Add pixel-level motion QA probes and a reviewable report; automated geometry warnings are not proof of visual quality.
- [x] Add bounded per-pixel PNG rest/activity comparison, optional eye/face ROI and unexpected transparent border metrics; expose through the CLI with deterministic tests and actual Hana frame evidence.
- [x] Add 2–60 ordered-frame RGB motion triage with optional ROI, expected control samples and intentional cut indices; flag candidate jumps and stalled intervals, verified against a real 13-frame Hana blink cycle.
- [ ] Generate a first-class visual contact sheet from the CLI and add head/neck seam-specific checks; pixel metrics must never auto-approve a rig.
- [ ] Wrap Rev2D's existing drawn closed-eye/mouth variants to reduce visible deformation artifacts without writing another mesh engine.

## Acceptance benchmark
Expand Down
69 changes: 69 additions & 0 deletions docs/PIXEL_QA.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Pixel-level visual QA (triage only)

**Status:** local PNG comparison implemented with deterministic RGBA decoding through the already-required FFmpeg executable. This is **not** a visual-approval model and not a perceptual similarity metric.

## CLI

Capture two frame images of exactly the same canvas and resolution. For a Rev2D avatar, use the engine's `render` command (review the current pinned backend help for flags) or the generated scene preview PNGs.

~~~sh
node dist/cli/index.js character qa compare outputs/rest.png outputs/eyes-closed.png
node dist/cli/index.js character qa compare outputs/rest.png outputs/eyes-closed.png --region 175,220,170,75
~~~

The optional region is an **integer** `x,y,width,height` rectangle within both images. Always use comparable camera, framing and character pose settings; do not compare resized images or a camera cut to an unchanged rest pose and interpret the result as articulation.

The output includes:

| Field | Meaning |
| --- | --- |
| `rgbMeanAbsoluteError` | Average absolute difference per RGB channel, in 0–255 intensity levels |
| `changedPixels` and `changedFraction` | Pixels whose RGB or alpha changed by more than 10 levels; fraction of comparable ROI pixels |
| `ignoredTransparentPixels` | ROI pixels where both samples have negligible alpha and RGB is visually undefined |
| `unexpectedTransparentBorderPixels` | Full-image perimeter pixels that were opaque in the reference and became nonopaque in the candidate |
| `region` | Coordinates actually evaluated for motion and RGB differences |

Inputs must be PNGs under 30 MB with dimensions 256–4096 pixels per side. Files must be inside the active workspace. PNGs with mismatched dimensions or invalid ROI are rejected rather than silently scaled. FFmpeg is invoked with fixed argument arrays, without a shell, and decoded RGBA buffers are bounded.

## Real-data probe (Hana, 2026-11-07)

A fresh local 512×640 Hana import was rendered at rest and again with both eye-open parameters set to 0, using the same native camera and an opaque background. The resulting source-versus-rest full-frame comparison gave `rgbMeanAbsoluteError = 0.0002`, `changedPixels = 0`, and `unexpectedTransparentBorderPixels = 0` for that specific imported test drawing.

For a 170×75 eye-region crop, rest versus closed-eyes produced `rgbMeanAbsoluteError = 33.5879`, `changedFraction = 0.364392`, and `unexpectedTransparentBorderPixels = 0`. These numbers establish **pixel activity**, not the quality of the drawing. Human visual inspection of the same review sheets still showed flat half-eye artwork and double contour artifacts in fades.

## Ordered-frame sequence QA (temporal motion)

Provide **2–60 PNGs in time order**. This does not infer sorting from filenames. The CLI analyzes adjacent frames, excludes declared intentional cuts from its local comparison baseline and reports unusually large changes as candidates for visual inspection.

~~~sh
node dist/cli/index.js character qa sequence outputs/f00.png outputs/f01.png outputs/f02.png outputs/f03.png --region 175,220,170,75
node dist/cli/index.js character qa sequence outputs/f00.png outputs/f01.png outputs/f02.png outputs/f03.png --region 175,220,170,75 --control-values 1,0.7,0.3,0
~~~

The optional `--control-values` list needs exactly one finite numeric control sample **per frame** and must describe a specific animated rig parameter (for example, `EyeLOpen`). When the control moves but rendered pixels barely change, the result is a `possibleMotionStalls` candidate. A static region, a clamped value, occlusion or an intentionally paused animation can also cause a flagged pair, so **visual review remains mandatory**.

For edited scenes, supply zero-based `--cut-after` indices to identify intended jumps: for example, `--cut-after 2` means the change between input frames 2 and 3 is an editorial cut and should not be treated as a surprise jump. The CLI reports the per-pair differences and candidate indices; it **never** returns a binary animation-quality verdict. The `firstToLast` measurement helps verify loop returns when the endpoints should match, but a deliberately non-looping scene need not return to start.

### Real 13-frame blink diagnosis (Hana, 2026-11-07)

The previously generated, intentionally unapproved Hana sprite trial was rendered locally at 512×640 with `EyeLOpen` and `EyeROpen` stepped together from 1 to 0 and back to 1 in 13 frames. A silent 30fps MP4 and manually assembled contact sheet were inspected under ignored `outputs/qa/temporal-blink-20261107/`. The new report showed:

| Metric | Observed |
| --- | --- |
| Adjacent pairs / frames | 12 / 13 |
| ROI | x 175, y 220, width 170, height 75 |
| Median adjacent RGB difference | 10.7044 / 255 |
| Potential abrupt pairs | none under generic outlier heuristic |
| Expected-control motion stalls | pairs 3→4 and 8→9 (0.166667 control delta, 0 RGB difference) |
| First versus last frame difference | exactly 0 |
| Video check | 512×640 H.264, 30 fps, 13 frames, 0.433333 seconds |

Rev2D's documented `eyes_half` variant intentionally remains fully visible over `EyeOpen` 0.3–0.6, explaining the two flat intervals; the tool cannot decide whether this artistic hold is acceptable. **Actual visual inspection** showed eye-outline ghosting at intermediate fade frames, even though the generic jump detector did not flag them. A smooth-looking statistic therefore does not imply smooth-looking animation. The generated variant art is experimental, locally ignored, not published and **not approved for use**.

## Limitations and next checks

- A different or badly deformed eye can produce a *higher* difference score; there is no universal minimum or maximum that certifies good animation.
- Closed-eyelid anatomy, visible tears, head/neck continuity, seam quality, motion cadence, and voice sync still require inspecting rendered frames/video.
- Frame-to-frame cuts, camera movement and background replacement must be identified semantically before interpreting pixel deltas.
- Thresholds should be calibrated per rig on diverse properly licensed artwork, not tuned to one Hana sample.
- This helper does not transmit artwork to an external model. It decodes PNGs locally using FFmpeg.
2 changes: 1 addition & 1 deletion docs/QUALITY_AUDIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Rev2D creates eye-white, iris, upper/lower lid, lash, crease and foreground-hair
## 6. Next milestones

1. Wrap Rev2D drawn-variant requests and imports as guarded PuppetControl CLI operations. Require explicit user/artwork permission and renewed review. No new mesh engine.
2. Add image-based QA probes for rest-pose identity, blink/gaze/mouth activity, and head/neck discontinuities. Quantitative reports are triage, not automatic visual approval.
2. **Pixel QA extended to ordered sequences:** 2–60 matching PNGs, optional eye/face ROI, declared hard-cut exemptions and actual parameter samples. The measured 512×640 Hana 13-frame blink probe at 30fps returned to its exact first image, but two intended-control intervals (3→4 and 8→9) yielded *identical* consecutive images, indicating a plateau. This is consistent with Rev2D's documented full-strength `eyes_half` sprite interval (`EyeOpen` 0.3–0.6), so it is **not** evidence of an unexpected renderer bug. The report flagged both intervals for human review, and no abrupt-change candidates were flagged. Manual inspection of the local contact sheet found overlapping outlines in intermediate blink frames (not captured by generic spike detection). See [PIXEL_QA.md](PIXEL_QA.md). Still **not** visually approved.
3. Build a diverse rights-cleared benchmark matrix: eye size, hair over face, accessories/glasses, different skin tones and line styles, opaque versus transparent backgrounds and constrained 3/4 views. Track success/failure on each, not just demos.
4. Only after a reliable single actor: improve body-pose transitions, synchronized multi-actor shots and user-facing authoring polish.

Expand Down
Loading
Loading