Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions .agents/skills/puppetcontrol/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,14 @@ Read repo-root AGENTS.md, docs/TECHNICAL_CONCEPT.md, and docs/CHARACTER_IMPORT.m
4. For optional image-model-generated landmarks, run `node dist/cli/index.js character landmarks propose <project-dir> --provider codex` only when the user accepts sending artwork to their configured Codex provider. The command produces a review candidate or diagnostic output, never an approved rig. Otherwise, for a manual agent task, run `node dist/cli/index.js character landmarks assist <project-dir>`. Open source.png, work/grid.png and work/landmark-agent-task.md. Fill the generated JSON proposal template with original source-pixel coordinates; do not guess from resized images. Read .cache/rev2d/docs/AVATAR.md.
5. Stage the proposal using `node dist/cli/index.js character landmarks stage <project-dir> --proposal <proposal.json>`. Rev2D validates the isolated candidate and returns a work/overlay.png. **Actually inspect the overlay** before `character landmarks accept <project-dir> --candidate <returned-dir> --reviewed`. Accepting replaces avatar.json with a backup and revokes all existing pose approvals. Never auto-pass --reviewed solely because a validator passed.
6. Build using node dist/cli/index.js character build <project-dir> (use --force if prior output exists). Inspect any geometryWarnings, particularly BODY_PIVOT_TOO_LOW and IRIS_LANDMARK_MISSING; these do not replace pixel inspection. Source/landmark/model/texture hashes are recorded in avatar-build.json and scene preflight rejects stale assets. Inspect every returned review/poses.png, turn.png, eyes.png and mouth.png. If incorrect, repair avatar.json and rebuild using --force; this resets the generated manifest and any edited scaffold.
7. A successful build intentionally leaves character.json with approved:false. A --force rebuild revokes approval *before* writing model/texture files, even if the rebuild later fails. Only after actual visual QA may an agent/user explicitly run `node dist/cli/index.js character approve <project-dir> --reviewed` (the flag is an acknowledgement, not evidence of review) to record neutral-pose approval. If there are torn eyelids, missing areas, extra contours or identity drift, LEAVE IT UNAPPROVED and report what needs correction.
8. For a reviewed reusable rig, use `node dist/cli/index.js scene new <new-scene.json> --character <character.json> --seconds 12 --fps 15` to create a strict editable scene. Reuse the same character file in a second scene without rebuilding artwork. The generated preview-scene.json is also editable. Use node dist/cli/index.js scene validate <scene.json> and scene plan before rendering. Preview PNG frames, inspect key timestamps, then render and inspect the MP4.
9. Within a compatible shared rig, semantic pose events may interpolate through validated parameter presets (the neutral-to-pointing demo uses shoulder and elbow controls). Distinct pose models require explicit scene camera cuts; an authored bridge is NOT implemented. Never silently crossfade meshes or call a body cut a smooth physical transition.
10. For speech, prepare WAV, install Rhubarb separately, and use node dist/cli/index.js audio align <voice.wav> --out <mouth.json> [--binary <rhubarb>]. A `scene new` voice track requires both `--voice` and the actual `--cues` file; no fabricated speech sync. Reference audio.path, audio.startMs and audio.cuesPath from the scene; review the actual mouth/audio timing.
11. Do not commit private character illustrations, full generated models, user voice recordings or secrets without checking redistribution rights. Demo:avatar generates pinned MIT-licensed sample artwork into ignored .cache.
12. State what actually rendered and which controls remain unsupported. Do not claim a flat PNG is automatically converted into production-quality animation without landmark review.
7. To test optional drawn eye/mouth variants on a built actor, request edit masks, locally supply reviewed `variants/<id>.png`, and explicitly import. With reviewed closed+half eye drawings, `character variants import <project> eyes_closed eyes_half --experimental-short-fade --reviewed-art` shortens fade windows but makes poses change faster and may still ghost; it is a visual experiment, never a default or automatic quality fix. The import revokes prior approval and refreshes provenance only after re-rendering review sheets.
8. Before approving a verified imported rig, run `node dist/cli/index.js character qa blink <project-dir>` to generate an ordered 25-frame actual eyelid transition contact sheet (`blink-eyes.png` and `blink-full.png`); inspect EyeOpen=0.19 and 0.69 and the real eye artwork seams. The QA report flags motion stalls but never passes visual approval.
9. A successful build intentionally leaves character.json with approved:false. A --force rebuild revokes approval *before* writing model/texture files, even if the rebuild later fails. Only after actual visual QA may an agent/user explicitly run `node dist/cli/index.js character approve <project-dir> --reviewed` (the flag is an acknowledgement, not evidence of review) to record neutral-pose approval. If there are torn eyelids, missing areas, extra contours or identity drift, LEAVE IT UNAPPROVED and report what needs correction.
10. For a reviewed reusable rig, use `node dist/cli/index.js scene new <new-scene.json> --character <character.json> --seconds 12 --fps 15` to create a strict editable scene. Reuse the same character file in a second scene without rebuilding artwork. The generated preview-scene.json is also editable. Use node dist/cli/index.js scene validate <scene.json> and scene plan before rendering. Preview PNG frames, inspect key timestamps, then render and inspect the MP4.
11. Within a compatible shared rig, semantic pose events may interpolate through validated parameter presets (the neutral-to-pointing demo uses shoulder and elbow controls). Distinct pose models require explicit scene camera cuts; an authored bridge is NOT implemented. Never silently crossfade meshes or call a body cut a smooth physical transition.
12. For speech, prepare WAV, install Rhubarb separately, and use node dist/cli/index.js audio align <voice.wav> --out <mouth.json> [--binary <rhubarb>]. For exact-byte continuity, optionally run `audio bind <voice.wav> --cues <mouth.json> --out <mouth.binding.json>` and include `--binding <mouth.binding.json>` in `scene new`; bound scenes fail validation/render if cue or voice bytes change. A `scene new` voice track requires both `--voice` and the actual `--cues` file; no fabricated speech sync. Reference audio.path, audio.startMs and audio.cuesPath from the scene; review the actual mouth/audio timing.
13. Do not commit private character illustrations, full generated models, user voice recordings or secrets without checking redistribution rights. Demo:avatar generates pinned MIT-licensed sample artwork into ignored .cache.
14. For animation QA, compare real ordered PNGs with `node dist/cli/index.js character qa sequence <f0.png> <f1.png> ... [--region x,y,w,h] [--control-values v0,v1,...] [--cut-after i,j]`. Supply sampled control values when investigating invisible or stalled motion; label known cuts. Inspect the PNGs/video and report possible ghosting even if the motion statistics look normal. The QA report is not an approval.
15. State what actually rendered and which controls remain unsupported. Do not claim a flat PNG is automatically converted into production-quality animation without landmark review.

Keep edits small and testable. Run npm run verify and inspect visual evidence for any animation/asset changes.
10 changes: 10 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,11 @@ The long-term goal: **generate artwork when necessary, animate and reuse it inde
- An opt-in `character approve <project> --reviewed` gate for visually inspected imported rigs; asset fingerprints and review sheets are checked before approval.
- `scene new` creates editable expression/look/dialogue scenes from an already approved rig without changing the character; voice requires genuine Rhubarb cues.
- Non-destructive MP4 export: output files are promoted only after successful FFmpeg encoding, with overwrite requiring explicit `--force`.
- Guarded **local drawn eye/mouth variant** request, status and import commands using Rev2D masks and sprites. Imported drawings revoke approval and require a fresh visual review; no image generation happens automatically.
- An optional **experimental** short-fade profile for imported half/closed eye drawings; avoids prolonged duplicate lash outlines at the cost of faster pose changes, revokes approval and regenerates review sheets.
- Local PNG pixel and ordered-frame QA (full frame or explicit region) reports RGB activity, unexpected transparency, candidate motion jumps and changed-control stalls; these metrics never auto-approve a rig.
- Import review can now render a 25-frame blink contact sheet focused on the actual eyelid transition boundaries, with pixel/motion diagnostics that never auto-approve a rig; see [Character Review](docs/CHARACTER_REVIEW.md).
- Optional SHA-256 Rhubarb cue bindings can detect when audio or mouth-cue bytes are replaced after alignment; a bound scene fails validation/render instead of silently using stale lip sync.
- Agent-readable CLI output, tests and CI.

**Not implemented yet:** fully automatic landmark placement or reliable one-click rigging, AI-generated arbitrary new pose rigs, convincing smooth bridges between unrelated meshes (especially crossed arms), automatic transcription/TTS, captions, B-roll and multiple actors. The arm-raise gesture is genuinely animated, but its artwork is still prototype-quality. Other pose families may require edited hard cuts. The pose manifest is capability metadata; visual poses become usable only after their rig files are generated, validated and inspected.
Expand All @@ -48,6 +53,10 @@ npm run demo:avatar # Import a freshly rendered flat PNG and create an unappr
npm run demo:first-video # Prepare an ignored, unapproved Hana fixture for review
# After actually reviewing the pinned Hana test fixture only:
# node scripts/demo-first-video.mjs --reviewed-sample
# For a built imported avatar, see docs/CHARACTER_IMPORT.md for drawn-variant request/import
# Compare captured PNG frames:
# node dist/cli/index.js character qa compare outputs/rest.png outputs/closed.png --region 175,220,170,75
# node dist/cli/index.js character qa sequence outputs/f00.png outputs/f01.png outputs/f02.png --control-values 1,0.5,0
~~~

The sample uses Rev2D's own **Hana** rig from the pinned MIT-licensed upstream checkout; generated character rig JSON is not checked into this repository. The derived example video is separately attributed and licensed under the original Rev2D terms.
Expand Down Expand Up @@ -102,6 +111,7 @@ Our native animation backend for the spike is **Rev2D**, pinned by the setup hel
- [Agent-guided landmark review and acceptance](docs/LANDMARK_AGENT_WORKFLOW.md)
- [Optional Codex vision inference: real-image results and limitations](docs/CODEX_VISION_LANDMARKS.md)
- [Deep quality and reliability audit with observed pixel measurements](docs/QUALITY_AUDIT.md)
- [Pixel-level visual QA (non-approval diagnostics)](docs/PIXEL_QA.md)
- [Architecture decision log](docs/DECISIONS.md)

Audio/lip-sync works with precomputed Rhubarb cues and a locally installed executable. The smooth pointing demo reuses the same face and deforms one rig continuously; crossed arms still need a more convincing bridge or intentional cut. Next: polish eyelid and mouth deformation by reusing Rev2D's reviewed drawn variants, expand automated image-quality probes and evaluate diverse source artwork. Do not confuse data validation with visual quality.
Expand Down
32 changes: 32 additions & 0 deletions docs/CHARACTER_IMPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,38 @@ node dist/cli/index.js scene render avatars/portrait/preview-scene.json --out ou

Scene validation also checks avatar-build.json against source.png, avatar.json, the built model and all generated textures. Changing any of them requires a new build and renewed review; stale files fail with AVATAR_ASSETS_STALE. Before approval, the scene intentionally fails with DEFAULT_POSE_NOT_APPROVED. The built model begins with one body pose family (neutral), not automatic pointing or crossed-arms artwork. Its provenance record covers that primary rig only: adding a different pose model to this imported actor is rejected until separate pose-model fingerprints are supported. Manually authored multi-pose characters remain available through the existing explicit rig workflow.

## Improve eyes and mouth with drawn variants (experimental)

If the automatically deformed eyelids or mouth look smeared, use **Rev2D's existing drawn sprites**. This does not automatically create beautiful art: someone must supply a carefully edited copy of the original PNG. The renderer falls back to mesh deformation for shapes without an accepted drawing.

~~~sh
node dist/cli/index.js character variants request avatars/portrait eyes_closed eyes_half mouth_a
node dist/cli/index.js character variants status avatars/portrait
~~~

The request produces `variant-requests/<id>.mask.png` and `variant-requests/<id>.prompt.txt` beside the original `source.png`. The mask's opaque black pixels must remain identical; only its editable, transparent area should be redrawn. Save each **full-size** corrected illustration as `variants/<id>.png` (for example `variants/eyes_closed.png`). A person or an image generator with **explicit artwork/privacy permission** may produce these PNGs; **PuppetControl's variant commands make no external image requests**.

After examining each candidate at full resolution, import only those exact IDs:

~~~sh
node dist/cli/index.js character variants import avatars/portrait eyes_closed eyes_half mouth_a --reviewed-art
node dist/cli/index.js character variants status avatars/portrait
~~~

For illustrations with **both** `eyes_closed` and `eyes_half` correctly drawn, there is an additional **opt-in experiment**:

~~~sh
node dist/cli/index.js character variants import avatars/portrait eyes_closed eyes_half --experimental-short-fade --reviewed-art
~~~

The default import preserves Rev2D's native fades. The experimental switch shortens the eye-sprite crossfade windows to `EyeOpen 0.18–0.20` (closed to half) and `0.68–0.70` (half to mesh). This reduces double outlines in some frames, **but makes the eyelids pop more quickly**, and ghosting remains possible within the short transition windows. It requires both drawn eye shapes on both sides and refuses unsupported/smiling-eye setups. A later import **without** the flag returns to the native generated fades. This is not a production-quality guarantee.

Import checks the drawing size, then delegates to Rev2D's mask, global shift/zoom and seam checks. It revokes every existing pose approval **before** touching model or texture assets; on success it re-renders `review/eyes.png`, `review/mouth.png` and `review/poses.png` and refreshes asset fingerprints. **It never auto-approves.** Inspect the output for face jumps, halos, duplicate lines and inconsistent colors, then explicitly reapprove in `character.json` if acceptable.

An upstream import can partially accept images before rejecting another. On error, the character remains unapproved and the old provenance may intentionally be stale; rebuild with `character build avatars/portrait --force` if necessary, inspect again, and only then retry. Do not assume that a successful import means the new drawings look good. Automatic image generation, reliable facial polish and smooth unrelated-pose bridges remain separate goals.

For a detailed eyelid/mouth correction procedure and a 25-frame blink contact sheet, see [CHARACTER_REVIEW.md](CHARACTER_REVIEW.md). Use `node dist/cli/index.js character qa blink avatars/portrait` before approval; it never modifies the rig or approves it. The original imported Hana trial still exhibits significant seams and is not acceptable character art.

## Reproducible built-in demonstration

~~~sh
Expand Down
Loading
Loading