Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions .agents/skills/puppetcontrol/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,12 @@ Read repo-root AGENTS.md, docs/TECHNICAL_CONCEPT.md, and docs/CHARACTER_IMPORT.m
4. For optional image-model-generated landmarks, run `node dist/cli/index.js character landmarks propose <project-dir> --provider codex` only when the user accepts sending artwork to their configured Codex provider. The command produces a review candidate or diagnostic output, never an approved rig. Otherwise, for a manual agent task, run `node dist/cli/index.js character landmarks assist <project-dir>`. Open source.png, work/grid.png and work/landmark-agent-task.md. Fill the generated JSON proposal template with original source-pixel coordinates; do not guess from resized images. Read .cache/rev2d/docs/AVATAR.md.
5. Stage the proposal using `node dist/cli/index.js character landmarks stage <project-dir> --proposal <proposal.json>`. Rev2D validates the isolated candidate and returns a work/overlay.png. **Actually inspect the overlay** before `character landmarks accept <project-dir> --candidate <returned-dir> --reviewed`. Accepting replaces avatar.json with a backup and revokes all existing pose approvals. Never auto-pass --reviewed solely because a validator passed.
6. Build using node dist/cli/index.js character build <project-dir> (use --force if prior output exists). Inspect any geometryWarnings, particularly BODY_PIVOT_TOO_LOW and IRIS_LANDMARK_MISSING; these do not replace pixel inspection. Source/landmark/model/texture hashes are recorded in avatar-build.json and scene preflight rejects stale assets. Inspect every returned review/poses.png, turn.png, eyes.png and mouth.png. If incorrect, repair avatar.json and rebuild using --force; this resets the generated manifest and any edited scaffold.
7. A successful build intentionally leaves character.json with approved:false. A --force rebuild revokes approval *before* writing model/texture files, even if the rebuild later fails. Only after actual visual QA should an agent/user mark the neutral pose approved. If there are torn eyelids, missing areas, extra contours or identity drift, LEAVE IT UNAPPROVED and report what needs correction.
8. The generated preview-scene.json is editable. Use node dist/cli/index.js scene validate <scene.json> and scene plan before rendering. Preview PNG frames, inspect key timestamps, then render and inspect the MP4.
9. Within a compatible shared rig, semantic pose events may interpolate through validated parameter presets (the neutral-to-pointing demo uses shoulder and elbow controls). Distinct pose models require explicit scene camera cuts; an authored bridge is NOT implemented. Never silently crossfade meshes or call a body cut a smooth physical transition.
10. For speech, prepare WAV, install Rhubarb separately, and use node dist/cli/index.js audio align <voice.wav> --out <mouth.json> [--binary <rhubarb>]. Reference audio.path, audio.startMs and audio.cuesPath from the scene; review the actual mouth/audio timing.
11. Do not commit private character illustrations, full generated models, user voice recordings or secrets without checking redistribution rights. Demo:avatar generates pinned MIT-licensed sample artwork into ignored .cache.
12. State what actually rendered and which controls remain unsupported. Do not claim a flat PNG is automatically converted into production-quality animation without landmark review.
7. To test optional drawn eye/mouth variants on a built actor, request edit masks, locally supply reviewed `variants/<id>.png`, and explicitly import. With reviewed closed+half eye drawings, `character variants import <project> eyes_closed eyes_half --experimental-short-fade --reviewed-art` shortens fade windows but makes poses change faster and may still ghost; it is a visual experiment, never a default or automatic quality fix. The import revokes prior approval and refreshes provenance only after re-rendering review sheets.
8. A successful build intentionally leaves character.json with approved:false. A --force rebuild revokes approval *before* writing model/texture files, even if the rebuild later fails. Only after actual visual QA should an agent/user mark the neutral pose approved. If there are torn eyelids, missing areas, extra contours or identity drift, LEAVE IT UNAPPROVED and report what needs correction.
9. The generated preview-scene.json is editable. Use node dist/cli/index.js scene validate <scene.json> and scene plan before rendering. Preview PNG frames, inspect key timestamps, then render and inspect the MP4.
10. Within a compatible shared rig, semantic pose events may interpolate through validated parameter presets (the neutral-to-pointing demo uses shoulder and elbow controls). Distinct pose models require explicit scene camera cuts; an authored bridge is NOT implemented. Never silently crossfade meshes or call a body cut a smooth physical transition.
11. For speech, prepare WAV, install Rhubarb separately, and use node dist/cli/index.js audio align <voice.wav> --out <mouth.json> [--binary <rhubarb>]. Reference audio.path, audio.startMs and audio.cuesPath from the scene; review the actual mouth/audio timing.
12. Do not commit private character illustrations, full generated models, user voice recordings or secrets without checking redistribution rights. Demo:avatar generates pinned MIT-licensed sample artwork into ignored .cache.
13. State what actually rendered and which controls remain unsupported. Do not claim a flat PNG is automatically converted into production-quality animation without landmark review.

Keep edits small and testable. Run npm run verify and inspect visual evidence for any animation/asset changes.
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ The long-term goal: **generate artwork when necessary, animate and reuse it inde
- Assisted single-PNG character onboarding through Rev2D landmarks, local rig build, review sheets, unapproved-by-default manifest, and starter scene.
- Agent-guided landmark proposals: source-hash-bound task, optional **tested** Codex image-based suggestion, isolated geometry validation, visual overlay and explicit review/acceptance that revokes old rig approval.
- Imported avatar asset provenance: source/spec/model/texture fingerprints checked by scene validation and render; untracked alternate imported pose models are refused. Force rebuild revokes approval before modifying art. Suspicious landmarks generate nonblocking geometry warnings.
- Guarded **local drawn eye/mouth variant** request, status and import commands using Rev2D masks and sprites. Imported drawings revoke approval and require a fresh visual review; no image generation happens automatically.
- An optional **experimental** short-fade profile for imported half/closed eye drawings; avoids prolonged duplicate lash outlines at the cost of faster pose changes, revokes approval and regenerates review sheets.
- Agent-readable CLI output, tests and CI.

**Not implemented yet:** fully automatic landmark placement or reliable one-click rigging, AI-generated arbitrary new pose rigs, convincing smooth bridges between unrelated meshes (especially crossed arms), automatic transcription/TTS, captions, B-roll and multiple actors. The arm-raise gesture is genuinely animated, but its artwork is still prototype-quality. Other pose families may require edited hard cuts. The pose manifest is capability metadata; visual poses become usable only after their rig files are generated, validated and inspected.
Expand All @@ -42,6 +44,7 @@ npm run demo:directed # Apply edited camera framing and gradual punch-in
npm run demo:gesture # Generate two-joint gesture rig and export edited 1080p demo
npm run demo:gesture-cycle # Render continuous raise/lower cycles without cuts
npm run demo:avatar # Import a freshly rendered flat PNG and create an unapproved rig + review sheets
# For a built imported avatar, see docs/CHARACTER_IMPORT.md for drawn-variant request/import
~~~

The sample uses Rev2D's own **Hana** rig from the pinned MIT-licensed upstream checkout; generated character rig JSON is not checked into this repository. The derived example video is separately attributed and licensed under the original Rev2D terms.
Expand Down
30 changes: 30 additions & 0 deletions docs/CHARACTER_IMPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,36 @@ node dist/cli/index.js scene render avatars/portrait/preview-scene.json --out ou

Scene validation also checks avatar-build.json against source.png, avatar.json, the built model and all generated textures. Changing any of them requires a new build and renewed review; stale files fail with AVATAR_ASSETS_STALE. Before approval, the scene intentionally fails with DEFAULT_POSE_NOT_APPROVED. The built model begins with one body pose family (neutral), not automatic pointing or crossed-arms artwork. Its provenance record covers that primary rig only: adding a different pose model to this imported actor is rejected until separate pose-model fingerprints are supported. Manually authored multi-pose characters remain available through the existing explicit rig workflow.

## Improve eyes and mouth with drawn variants (experimental)

If the automatically deformed eyelids or mouth look smeared, use **Rev2D's existing drawn sprites**. This does not automatically create beautiful art: someone must supply a carefully edited copy of the original PNG. The renderer falls back to mesh deformation for shapes without an accepted drawing.

~~~sh
node dist/cli/index.js character variants request avatars/portrait eyes_closed eyes_half mouth_a
node dist/cli/index.js character variants status avatars/portrait
~~~

The request produces `variant-requests/<id>.mask.png` and `variant-requests/<id>.prompt.txt` beside the original `source.png`. The mask's opaque black pixels must remain identical; only its editable, transparent area should be redrawn. Save each **full-size** corrected illustration as `variants/<id>.png` (for example `variants/eyes_closed.png`). A person or an image generator with **explicit artwork/privacy permission** may produce these PNGs; **PuppetControl's variant commands make no external image requests**.

After examining each candidate at full resolution, import only those exact IDs:

~~~sh
node dist/cli/index.js character variants import avatars/portrait eyes_closed eyes_half mouth_a --reviewed-art
node dist/cli/index.js character variants status avatars/portrait
~~~

For illustrations with **both** `eyes_closed` and `eyes_half` correctly drawn, there is an additional **opt-in experiment**:

~~~sh
node dist/cli/index.js character variants import avatars/portrait eyes_closed eyes_half --experimental-short-fade --reviewed-art
~~~

The default import preserves Rev2D's native fades. The experimental switch shortens the eye-sprite crossfade windows to `EyeOpen 0.18–0.20` (closed to half) and `0.68–0.70` (half to mesh). This reduces double outlines in some frames, **but makes the eyelids pop more quickly**, and ghosting remains possible within the short transition windows. It requires both drawn eye shapes on both sides and refuses unsupported/smiling-eye setups. A later import **without** the flag returns to the native generated fades. This is not a production-quality guarantee.

Import checks the drawing size, then delegates to Rev2D's mask, global shift/zoom and seam checks. It revokes every existing pose approval **before** touching model or texture assets; on success it re-renders `review/eyes.png`, `review/mouth.png` and `review/poses.png` and refreshes asset fingerprints. **It never auto-approves.** Inspect the output for face jumps, halos, duplicate lines and inconsistent colors, then explicitly reapprove in `character.json` if acceptable.

An upstream import can partially accept images before rejecting another. On error, the character remains unapproved and the old provenance may intentionally be stale; rebuild with `character build avatars/portrait --force` if necessary, inspect again, and only then retry. Do not assume that a successful import means the new drawings look good. Automatic image generation, reliable facial polish and smooth unrelated-pose bridges remain separate goals.

## Reproducible built-in demonstration

~~~sh
Expand Down
4 changes: 3 additions & 1 deletion docs/IMPLEMENTATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,9 @@
- [x] Revoke existing approval before any --force build mutates Rev2D outputs, and fail closed on stale source/spec/model/texture fingerprints.
- [x] Eliminate reproduced black edge artifacts caused by alpha-zero warped pixels in H.264: composite frames onto scene background, with a 60-frame before/after comparison.
- [ ] Add pixel-level motion QA probes and a reviewable report; automated geometry warnings are not proof of visual quality.
- [ ] Wrap Rev2D's existing drawn closed-eye/mouth variants to reduce visible deformation artifacts without writing another mesh engine.
- [x] Wrap local Rev2D drawn eye/mouth variant request/status/import with explicit reviewed-art selection, approval revocation before import, post-import provenance refresh, and new review sheets. A real Hana avatar produced correct edit masks; unchanged fake drawings were rejected.
- [x] Test and expose an opt-in, validated short eye-sprite fade as a local derived-model experiment. Compare 13 actual Hana blink frames with native, immediate-switch and shortened fades: shorter fades visibly reduce double lashes at sampled midpoints, but introduce faster pose changes; never auto-approve.
- [ ] Produce and visually approve high-quality closed/half eye and mouth artwork; demonstrate seamless real rendered transitions across full blink playback and multiple character styles. The local proof drawing stays unapproved.

## Acceptance benchmark

Expand Down
4 changes: 2 additions & 2 deletions docs/QUALITY_AUDIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,8 @@ Rev2D creates eye-white, iris, upper/lower lid, lash, crease and foreground-hair

## 6. Next milestones

1. Wrap Rev2D drawn-variant requests and imports as guarded PuppetControl CLI operations. Require explicit user/artwork permission and renewed review. No new mesh engine.
2. Add image-based QA probes for rest-pose identity, blink/gaze/mouth activity, and head/neck discontinuities. Quantitative reports are triage, not automatic visual approval.
1. **Guarded local drawn-variant CLI added:** request/status/import supported by pinned Rev2D. Request masks for eyes_closed, eyes_half and mouth_a on a fresh 512×640 Hana import were generated and visually inspected. Importing an unchanged eyes_closed image correctly failed upstream with code `unchanged` and did not approve the character. A separate *throwaway* Hana project then accepted locally hand-procedurally-redrawn eyes_closed and eyes_half artwork (both with outsideDiff=0 and nearDiff=0); approval remained false. Actual eyes review frames showed the closed-eye lashes more continuous than the original mesh deformation, but a faint lower contour, too-flat half-open eyelids and double contours during fades remained. **These test drawings are not acceptable character art**, are only in ignored local .cache, and are not redistributed. No AI provider was used and no arbitrary-character quality claim follows.
2. **Experimental fade mitigation tested:** a 13-frame 30fps Hana blink was rendered with native opacity blending, an immediate sprite switch and a shortened fade. The opt-in `--experimental-short-fade` importer now produces the shorter blend windows after Rev2D validates the derived model, regenerates review sheets and refreshes fingerprints. At sampled `EyeOpen=0.667`, the guarded import produced the exact same PNG hash as the clean short-fade experiment; the native frame visibly had double eye outlines. The short fade changes pose faster, and can still overlap shapes within the narrow transition windows. The rig remains **unapproved**. Add image-based QA probes for rest-pose identity, blink/gaze/mouth activity, and head/neck discontinuities. Quantitative reports are triage, not automatic visual approval.
3. Build a diverse rights-cleared benchmark matrix: eye size, hair over face, accessories/glasses, different skin tones and line styles, opaque versus transparent backgrounds and constrained 3/4 views. Track success/failure on each, not just demos.
4. Only after a reliable single actor: improve body-pose transitions, synchronized multi-actor shots and user-facing authoring polish.

Expand Down
7 changes: 6 additions & 1 deletion src/character/avatar.ts
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,12 @@ export async function runRev2D(repo: string, args: string[], timeout = 120_000):
// Return the upstream validator's precise diagnostics rather than hiding them.
let output: CommandOutput = {};
try { output = JSON.parse(failure.stdout ?? "{}") as CommandOutput; } catch { /* not JSON */ }
const diagnosticText = output.diagnostics ? JSON.stringify(output.diagnostics).slice(0, 3500) : "";
const rejected = Array.isArray(output.checks) ? output.checks
.filter((value) => !!value && typeof value === "object" && (value as { ok?: boolean }).ok === false)
.map((value) => ({ id: value.id, code: value.code, reason: value.reason, hint: value.hint })) : [];
const diagnosticText = Array.isArray(output.diagnostics) && output.diagnostics.length
? JSON.stringify(output.diagnostics).slice(0, 3500)
: rejected.length ? JSON.stringify(rejected).slice(0, 3500) : "";
throw new AvatarWorkflowError(
"REV2D_AVATAR_FAILED",
[diagnosticText, failure.stderr, failure.message].filter(Boolean).join("\n").slice(0, 5000),
Expand Down
Loading
Loading