Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .agents/skills/puppetcontrol/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ Read repo-root AGENTS.md, docs/TECHNICAL_CONCEPT.md, and docs/CHARACTER_IMPORT.m
4. For optional image-model-generated landmarks, run `node dist/cli/index.js character landmarks propose <project-dir> --provider codex` only when the user accepts sending artwork to their configured Codex provider. The command produces a review candidate or diagnostic output, never an approved rig. Otherwise, for a manual agent task, run `node dist/cli/index.js character landmarks assist <project-dir>`. Open source.png, work/grid.png and work/landmark-agent-task.md. Fill the generated JSON proposal template with original source-pixel coordinates; do not guess from resized images. Read .cache/rev2d/docs/AVATAR.md.
5. Stage the proposal using `node dist/cli/index.js character landmarks stage <project-dir> --proposal <proposal.json>`. Rev2D validates the isolated candidate and returns a work/overlay.png. **Actually inspect the overlay** before `character landmarks accept <project-dir> --candidate <returned-dir> --reviewed`. Accepting replaces avatar.json with a backup and revokes all existing pose approvals. Never auto-pass --reviewed solely because a validator passed.
6. Build using node dist/cli/index.js character build <project-dir> (use --force if prior output exists). Inspect any geometryWarnings, particularly BODY_PIVOT_TOO_LOW and IRIS_LANDMARK_MISSING; these do not replace pixel inspection. Source/landmark/model/texture hashes are recorded in avatar-build.json and scene preflight rejects stale assets. Inspect every returned review/poses.png, turn.png, eyes.png and mouth.png. If incorrect, repair avatar.json and rebuild using --force; this resets the generated manifest and any edited scaffold.
7. A successful build intentionally leaves character.json with approved:false. A --force rebuild revokes approval *before* writing model/texture files, even if the rebuild later fails. Only after actual visual QA should an agent/user mark the neutral pose approved. If there are torn eyelids, missing areas, extra contours or identity drift, LEAVE IT UNAPPROVED and report what needs correction.
8. The generated preview-scene.json is editable. Use node dist/cli/index.js scene validate <scene.json> and scene plan before rendering. Preview PNG frames, inspect key timestamps, then render and inspect the MP4.
7. A successful build intentionally leaves character.json with approved:false. A --force rebuild revokes approval *before* writing model/texture files, even if the rebuild later fails. Only after actual visual QA may an agent/user explicitly run `node dist/cli/index.js character approve <project-dir> --reviewed` (the flag is an acknowledgement, not evidence of review) to record neutral-pose approval. If there are torn eyelids, missing areas, extra contours or identity drift, LEAVE IT UNAPPROVED and report what needs correction.
8. For a reviewed reusable rig, use `node dist/cli/index.js scene new <new-scene.json> --character <character.json> --seconds 12 --fps 15` to create a strict editable scene. Reuse the same character file in a second scene without rebuilding artwork. The generated preview-scene.json is also editable. Use node dist/cli/index.js scene validate <scene.json> and scene plan before rendering. Preview PNG frames, inspect key timestamps, then render and inspect the MP4.
9. Within a compatible shared rig, semantic pose events may interpolate through validated parameter presets (the neutral-to-pointing demo uses shoulder and elbow controls). Distinct pose models require explicit scene camera cuts; an authored bridge is NOT implemented. Never silently crossfade meshes or call a body cut a smooth physical transition.
10. For speech, prepare WAV, install Rhubarb separately, and use node dist/cli/index.js audio align <voice.wav> --out <mouth.json> [--binary <rhubarb>]. Reference audio.path, audio.startMs and audio.cuesPath from the scene; review the actual mouth/audio timing.
10. For speech, prepare WAV, install Rhubarb separately, and use node dist/cli/index.js audio align <voice.wav> --out <mouth.json> [--binary <rhubarb>]. A `scene new` voice track requires both `--voice` and the actual `--cues` file; no fabricated speech sync. Reference audio.path, audio.startMs and audio.cuesPath from the scene; review the actual mouth/audio timing.
11. Do not commit private character illustrations, full generated models, user voice recordings or secrets without checking redistribution rights. Demo:avatar generates pinned MIT-licensed sample artwork into ignored .cache.
12. State what actually rendered and which controls remain unsupported. Do not claim a flat PNG is automatically converted into production-quality animation without landmark review.

Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ The long-term goal: **generate artwork when necessary, animate and reuse it inde
- Assisted single-PNG character onboarding through Rev2D landmarks, local rig build, review sheets, unapproved-by-default manifest, and starter scene.
- Agent-guided landmark proposals: source-hash-bound task, optional **tested** Codex image-based suggestion, isolated geometry validation, visual overlay and explicit review/acceptance that revokes old rig approval.
- Imported avatar asset provenance: source/spec/model/texture fingerprints checked by scene validation and render; untracked alternate imported pose models are refused. Force rebuild revokes approval before modifying art. Suspicious landmarks generate nonblocking geometry warnings.
- An opt-in `character approve <project> --reviewed` gate for visually inspected imported rigs; asset fingerprints and review sheets are checked before approval.
- `scene new` creates editable expression/look/dialogue scenes from an already approved rig without changing the character; voice requires genuine Rhubarb cues.
- Non-destructive MP4 export: output files are promoted only after successful FFmpeg encoding, with overwrite requiring explicit `--force`.
- Agent-readable CLI output, tests and CI.

**Not implemented yet:** fully automatic landmark placement or reliable one-click rigging, AI-generated arbitrary new pose rigs, convincing smooth bridges between unrelated meshes (especially crossed arms), automatic transcription/TTS, captions, B-roll and multiple actors. The arm-raise gesture is genuinely animated, but its artwork is still prototype-quality. Other pose families may require edited hard cuts. The pose manifest is capability metadata; visual poses become usable only after their rig files are generated, validated and inspected.
Expand All @@ -42,6 +45,9 @@ npm run demo:directed # Apply edited camera framing and gradual punch-in
npm run demo:gesture # Generate two-joint gesture rig and export edited 1080p demo
npm run demo:gesture-cycle # Render continuous raise/lower cycles without cuts
npm run demo:avatar # Import a freshly rendered flat PNG and create an unapproved rig + review sheets
npm run demo:first-video # Prepare an ignored, unapproved Hana fixture for review
# After actually reviewing the pinned Hana test fixture only:
# node scripts/demo-first-video.mjs --reviewed-sample
~~~

The sample uses Rev2D's own **Hana** rig from the pinned MIT-licensed upstream checkout; generated character rig JSON is not checked into this repository. The derived example video is separately attributed and licensed under the original Rev2D terms.
Expand All @@ -54,6 +60,8 @@ The sample uses Rev2D's own **Hana** rig from the pinned MIT-licensed upstream c

[Watch the reversible gesture demo](docs/media/hana-gesture-cycle.mp4) (5 seconds, silent 30 fps; the same arm moves between neutral and pointing four times with no cuts).

For the first reproducible **one-character video workflow**, see [First Video](docs/FIRST_VIDEO.md). With an already reviewed rig, run `character approve <project> --reviewed` for provenance-tracked imported characters, then `scene new <path> --character <character.json>`, `scene validate` and `scene render`. Create another scene with the same character path to reuse its rig, without regenerating artwork. To attach your own voice, use a real WAV and aligned Rhubarb cue JSON; no TTS is implied.

Run a custom scene through the CLI:

~~~sh
Expand Down
2 changes: 1 addition & 1 deletion docs/CHARACTER_IMPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ This revokes prior approval **before** the rendering backend touches the rig, an

Inspect every review sheet for torn eye closures, halos, doubled contours, missing hair/skin, incorrect mouth shapes, and awkward head movement. Correct the landmark source and rebuild as needed.

When the rendered rig is genuinely acceptable, change the neutral pose family's approved field to true in character.json. Only then:
When the rendered rig is genuinely acceptable, run `node dist/cli/index.js character approve avatars/portrait --reviewed`. This requires four existing review PNGs and a current source/spec/model/texture provenance record. The flag acknowledges review but cannot prove that someone genuinely inspected the artwork; never pass it automatically for user images. Only then:

~~~sh
node dist/cli/index.js scene validate avatars/portrait/preview-scene.json
Expand Down
103 changes: 103 additions & 0 deletions docs/FIRST_VIDEO.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# First usable video: two scenes, one reviewed rig

This page documents the smallest truthful PuppetControl flow that produces real MP4 files while reusing one already-reviewed Rev2D actor.

The integration demo uses the pinned MIT-licensed Rev2D **Hana** sample only. It never approves user artwork. Its approval marker lives in an ignored .cache/first-video-demo/... fixture and is created only after an explicit --reviewed-sample flag.

## Prerequisites

Use Node.js 22 or newer and install the repository dependencies. FFmpeg and FFprobe must be available on PATH.

Prepare the pinned rig backend and build the current CLI:

npm run rig:setup
npm run build

The expected Rev2D commit is f144581cdf1f12d781379c4df90d262b8c2dfa3d.

## Reproduce the two-video Hana demo

First prepare an **unapproved** ignored fixture and a PNG for visual review:

node scripts/demo-first-video.mjs

The command prints a path such as:

.cache/first-video-demo/run-XXXXXX/hana-review.png

Open that PNG and inspect the sample. PuppetControl cannot technically verify that a human actually reviewed or consented to an asset.

Only for this pinned Hana QA fixture, explicitly continue with:

node scripts/demo-first-video.mjs --reviewed-sample

That second command creates a **separate disposable copy** of the same pinned Hana sample. Its flag records acknowledgement for a test fixture, not verified review of that newly created copy. It uses the current CLI scene new, scene validate, and scene render commands to create two separate 5-second scenes that both point at the same copied Hana rig. It then runs the FFprobe acceptance checker on both MP4s.

Each run is isolated under ignored .cache/first-video-demo/run-XXXXXX/. Expected outputs are:

outputs/hana-scene-a.mp4
outputs/hana-scene-b.mp4

Both videos are silent H.264 MP4s at 512x640, 15 fps, 75 frames, and 5 seconds. The scenes differ in supported expression/look timing while keeping the same actor and rig bytes.

The checker validates container/stream metadata, dimensions, fps, duration, frame count, expected lack of audio, and availability of FFmpeg/FFprobe. That is **not** a visual-quality approval; inspect actual frames/video as part of QA.

## Use your own approved rig

The scene new command is for an actor that has already passed the project's real review/approval workflow. It refuses an unapproved default pose and does not auto-approve artwork. For a single-image Rev2D import, after genuinely inspecting all four review sheets, acknowledge approval explicitly with:

node dist/cli/index.js character approve path/to/avatar-project --reviewed

This validates source/landmark/rig/texture fingerprints and presence of review sheets before recording approval. It cannot technically verify that a human looked at them. Do not script approval for unreviewed user art.

After `npm run build`, create an editable scene. Commands below are each a single line so they work in **PowerShell**, Command Prompt, bash and other supported shells:

node dist/cli/index.js scene new .cache/my-video/scene-a.json --character path/to/character.json --id scene-a --seconds 5 --fps 15 --size 512x640

Edit only events that the character actually supports, then validate and render:

node dist/cli/index.js scene validate .cache/my-video/scene-a.json
node dist/cli/index.js scene render .cache/my-video/scene-a.json --out .cache/my-video/scene-a.mp4

For a second scene, run scene new again with a different output scene path and ID while keeping the same --character path. That reuses the same approved rig rather than rebuilding or regenerating the actor.

## Optional WAV + Rhubarb cues

Voice is optional. PuppetControl does not synthesize speech or pretend that lip sync exists when cues are missing.

If Rhubarb is installed, align a WAV first:

node dist/cli/index.js audio align path/to/voice.wav --out .cache/my-video/mouth.json --binary rhubarb

Then create the scene with both the WAV and cue JSON:

node dist/cli/index.js scene new .cache/my-video/voiced.json --character path/to/character.json --id voiced --seconds 5 --fps 15 --size 512x640 --voice path/to/voice.wav --cues .cache/my-video/mouth.json

The current first-video path requires --voice and --cues together. Validate and render the resulting scene normally. A real 12-second test using local nonredistributed voice input successfully generated 36 Rhubarb cues and a 180-frame H.264/AAC MP4 at 512×640, 15 FPS. The audio recording, cues and MP4 are intentionally kept only in ignored .cache/first-video-demo/qa-12s-voiced/. This proves file/cue/mux integration, not perceptual lip-sync quality on every speaker.

## Non-destructive rendering

By default, PuppetControl refuses to replace an existing MP4 at the chosen output path. The renderer writes to a unique temporary MP4 and publishes it only after FFmpeg successfully completes. To explicitly re-render over the same output path after editing a scene, use `--force`:

node dist/cli/index.js scene render .cache/my-video/scene-a.json --out .cache/my-video/scene-a.mp4 --force

For ordinary revisions, prefer a new `--out` name so the prior render remains available. Scene IDs by default follow the scene filename, reducing collisions between different scenes.

## Verify an exported MP4

For a 5-second 15-fps 512x640 silent export:

node scripts/check-first-video-video.mjs .cache/my-video/scene-a.mp4 --fps 15 --width 512 --height 640 --frames 75 --video-codec h264

For a voiced export, add --audio --audio-codec aac.

## Current limitations

- The first-video flow is intentionally one reviewed reusable actor; it is not a reliable one-click importer or automatic approval system.
- Approval state is a technical gate, not proof that a human reviewed the art or that the creator has rights/consent.
- The Hana --reviewed-sample path is a test-only exception isolated under ignored .cache; it must not be copied into production approval logic.
- Voice requires an existing WAV plus valid Rhubarb cue JSON. There is no automatic transcription or TTS in this flow.
- Current rendering supports at most one audio track and the first-video scaffold is aimed at one actor.
- Smooth arbitrary between-pose bridges are not implemented. Do not present hard cuts or unrelated-rig switches as fluid motion.
- Metadata acceptance with FFprobe does not establish animation quality. Inspect the rendered frames/video.
- Native rig rendering and presentation/output scaling are separate concerns; use dimensions that the current renderer has actually been tested with.
15 changes: 15 additions & 0 deletions docs/IMPLEMENTATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,21 @@
- [ ] Add pixel-level motion QA probes and a reviewable report; automated geometry warnings are not proof of visual quality.
- [ ] Wrap Rev2D's existing drawn closed-eye/mouth variants to reduce visible deformation artifacts without writing another mesh engine.

## First video milestone — reproducible usable slice (2026-11-07)

This milestone is deliberately **narrower** than M5's universal single-image promise. The verified workflow uses one previously reviewed Rev2D character, not an arbitrary one-click PNG rig. Commands and proof: [FIRST_VIDEO.md](FIRST_VIDEO.md).

- [x] Reuse an approved rig in two independently authored 5-second scenes, without changing character model, texture or manifest.
- [x] Create strict editable scene files via `scene new`, instead of hand-editing TypeScript; reject unsupported events and unapproved characters.
- [x] Add an explicit `character approve --reviewed` gate for one provenance-tracked imported rig with required review sheets. Software cannot prove human review, so this is a record of an explicit acknowledgement, never an automatic decision.
- [x] Validate the actual 75-frame, 15fps, 512×640 H.264 output of both scenes using FFprobe and visually inspect representative frames.
- [x] Render an additional 12-second, 180-frame H.264/AAC sample using 36 real Rhubarb cues from a local test WAV; verify both streams and inspect representative frames. Local audio and video are ignored, not published.
- [x] Prevent silent overwrite of existing MP4s by encoding to a temporary sibling file; `--force` explicitly replaces only after successful encoding.
- [x] Make `scene validate` distinguish planning-valid bridge/caption/motion documents from scenes the current renderer can actually export, with actionable errors.
- [ ] Achieve visibly polished eye/mouth deformation and user-friendly assisted rig correction on **diverse** artwork; the imported Hana avatar previously showed artifacts and remains unapproved.
- [ ] Verify acceptable lip synchronization with multiple natural speakers, and bind Rhubarb cue data to its source-audio fingerprint.
- [ ] Produce a convincing, reproducible 10–15s video **using an independently imported and genuinely approved custom illustration** without developer intervention. The 12s test so far uses a pre-rigged licensed sample.

## Acceptance benchmark

A single 45-second 1080p/30 video with human/AI voice, four pose families (three minimum), convincing continuous head/face movement, visually acceptable lip-sync, smooth or intentionally hidden pose changes, consistent identity, and a successful second export after a one-line expression edit.
Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
"demo:gesture-cycle": "npm run demo:gesture:rig && tsx src/cli/index.ts scene render examples/scenes/hana-gesture-cycle.json",
"demo:benchmark": "npm run demo:gesture:rig && tsx src/cli/index.ts scene render examples/scenes/hana-45s-benchmark.json",
"demo:avatar": "npm run build && node scripts/demo-avatar-import.mjs",
"demo:first-video": "npm run build && node scripts/demo-first-video.mjs",
"test": "vitest run",
"typecheck": "tsc --noEmit",
"build": "tsc",
Expand Down
Loading
Loading