Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .agents/skills/puppetcontrol/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Read repo-root AGENTS.md, docs/TECHNICAL_CONCEPT.md, and docs/CHARACTER_IMPORT.m
12. For speech, prepare WAV, install Rhubarb separately, and use node dist/cli/index.js audio align <voice.wav> --out <mouth.json> [--binary <rhubarb>]. For exact-byte continuity, optionally run `audio bind <voice.wav> --cues <mouth.json> --out <mouth.binding.json>` and include `--binding <mouth.binding.json>` in `scene new`; bound scenes fail validation/render if cue or voice bytes change. A `scene new` voice track requires both `--voice` and the actual `--cues` file; no fabricated speech sync. Reference audio.path, audio.startMs and audio.cuesPath from the scene; review the actual mouth/audio timing.
13. Do not commit private character illustrations, full generated models, user voice recordings or secrets without checking redistribution rights. Demo:avatar generates pinned MIT-licensed sample artwork into ignored .cache.
14. For animation QA, compare real ordered PNGs with `node dist/cli/index.js character qa sequence <f0.png> <f1.png> ... [--region x,y,w,h] [--control-values v0,v1,...] [--cut-after i,j]`. Supply sampled control values when investigating invisible or stalled motion; label known cuts. Inspect the PNGs/video and report possible ghosting even if the motion statistics look normal. The QA report is not an approval.
15. State what actually rendered and which controls remain unsupported. Do not claim a flat PNG is automatically converted into production-quality animation without landmark review.
15. Verified imported rigs and explicitly bound voice tracks are now consumed from fresh per-render copies, hashed against recorded inputs; see `docs/RENDER_INTEGRITY.md`. This is not protection against a malicious same-user process modifying temporary files, nor proof of visual quality.
16. State what actually rendered and which controls remain unsupported. Do not claim a flat PNG is automatically converted into production-quality animation without landmark review.

Keep edits small and testable. Run npm run verify and inspect visual evidence for any animation/asset changes.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ The long-term goal: **generate artwork when necessary, animate and reuse it inde
- An opt-in `character approve <project> --reviewed` gate for visually inspected imported rigs; asset fingerprints and review sheets are checked before approval.
- `scene new` creates editable expression/look/dialogue scenes from an already approved rig without changing the character; voice requires genuine Rhubarb cues.
- Non-destructive MP4 export: output files are promoted only after successful FFmpeg encoding, with overwrite requiring explicit `--force`.
- [Verified render inputs](docs/RENDER_INTEGRITY.md): imported rig/texture assets and bound voice/cues are copied and rehashed before subprocess use; fresh frame directories and explicit-root path checks reduce accidental file-replacement and symlink risks.
- Guarded **local drawn eye/mouth variant** request, status and import commands using Rev2D masks and sprites. Imported drawings revoke approval and require a fresh visual review; no image generation happens automatically.
- An optional **experimental** short-fade profile for imported half/closed eye drawings; avoids prolonged duplicate lash outlines at the cost of faster pose changes, revokes approval and regenerates review sheets.
- Local PNG pixel and ordered-frame QA (full frame or explicit region) reports RGB activity, unexpected transparency, candidate motion jumps and changed-control stalls; these metrics never auto-approve a rig.
Expand Down
2 changes: 2 additions & 0 deletions docs/FIRST_VIDEO.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,8 @@ By default, PuppetControl refuses to replace an existing MP4 at the chosen outpu

For ordinary revisions, prefer a new `--out` name so the prior render remains available. Scene IDs by default follow the scene filename, reducing collisions between different scenes.

For the filesystem and SHA-256 guarantees available during rendering (and their limits), see [RENDER_INTEGRITY.md](RENDER_INTEGRITY.md). Imported art and texture snapshots use a fresh render root; only scenes with `--binding` get a verified, private voice/cue copy before FFmpeg.

## Verify an exported MP4

For a 5-second 15-fps 512x640 silent export:
Expand Down
1 change: 1 addition & 0 deletions docs/IMPLEMENTATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ This milestone is deliberately **narrower** than M5's universal single-image pro
- [x] Validate the actual 75-frame, 15fps, 512×640 H.264 output of both scenes using FFprobe and visually inspect representative frames.
- [x] Render an additional 12-second, 180-frame H.264/AAC sample using 36 real Rhubarb cues from a local test WAV; verify both streams and inspect representative frames. Local audio and video are ignored, not published.
- [x] Prevent silent overwrite of existing MP4s by encoding to a temporary sibling file; `--force` explicitly replaces only after successful encoding.
- [x] Add verified per-render imported rig/textures and exact-byte-bound voice/cues copies, fresh frame roots, and explicit-root scene input confinement. Local QA produced a 45-frame silent imported-rig test export and a 45-frame H.264/AAC bound-voice export; this is not an OS sandbox against malicious concurrent file mutation.
- [x] Make `scene validate` distinguish planning-valid bridge/caption/motion documents from scenes the current renderer can actually export, with actionable errors.
- [ ] Achieve visibly polished eye/mouth deformation and user-friendly assisted rig correction on **diverse** artwork; the imported Hana avatar previously showed artifacts and remains unapproved.
- [x] Add optional exact-byte SHA-256 Rhubarb cue binding sidecars; verified bound first-video scene and live H.264/AAC render. Sidecars are opt-in, and legacy scenes remain supported.
Expand Down
36 changes: 36 additions & 0 deletions docs/RENDER_INTEGRITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Verified render inputs and filesystem boundaries

PuppetControl is a **local-first authoring CLI**. It processes user art and files supplied by agents. These safeguards address accidental source edits, ordinary file replacement between checks, and symlinked output destinations. They are **not a sandbox** against another process running with the same user permissions and deliberately modifying the private render snapshot during rendering.

## Imported-character rig integrity

After the normal imported-avatar SHA-256 build-record verification, `scene render` now copies the approved `source.png`, `avatar.json`, rig model and all recorded model textures into a fresh, ignored `verified-actor-assets/` directory under its unique frame-output root. Each copied file is SHA-256 checked against the existing build record. The Rev2D adapter reads the **copied** model and textures, never the original files during that render. If any bytes change before or during copying, the copy fails verification.

This closes the ordinary gap where a rig could be checked at scene-load time and replaced at its original path before Rev2D opened it. The imported test fixture must still be *genuinely* visually reviewed before user approval; an integrity record is not an aesthetic certification or an approval signature.

## Bound voice/cue integrity

`audio bind` now parses **the exact cue bytes** that it hashes. When a scene has `audio.bindingPath`, `scene render` verifies the sidecar, copies WAV and cue JSON into a fresh ignored `verified-speech/` directory, hashes both copied inputs against the binding and parses the **copied** cue JSON. Rhubarb tracks use those cues and FFmpeg receives the **copied audio path**.

This prevents ordinary source edits during an export from mixing new audio with an old cue file. Earlier scenes without an explicit binding remain supported and **do not** gain this protection.

## Render output and explicit roots

- Frame output must be **new**, rather than an existing directory that might contain old frames. A symlink/junction escape through its existing ancestors is rejected before rendering.
- The subprocess's list of frame names must match flat `frame_0000.png`-style names; a path-like name or traversal is rejected.
- `loadAndValidateScene(scene, { root })` now checks the scene, actors, and **all** audio, cue, and binding paths against the same realpath-aware root, returning diagnostics rather than throwing on escaped paths.
- The existing MP4 exporter still writes a unique temporary sibling MP4 and refuses overwrite without explicit `--force`.

## Reproduction and evidence (2026-11-07)

Run `npm run verify` to execute regression tests for copied asset integrity, path traversal, root-scoped loading, voice swaps, and guarded frame directories. For a locally installed pinned Rev2D backend and available FFmpeg, the actual video exports were separately exercised with two **ignored QA fixtures**:

- `.cache/render-snapshot-qa-20261107-1800/qa-snapshot.mp4`: 3 seconds, 15 FPS, 512×640, 45 H.264 frames, silent. **The test copy alone** has an explicit forced approval marker to exercise the renderer. The underlying imported artwork was not user-approved and its original manifest remains unapproved. Do not distribute or use the test marker in normal workflows.
- `.cache/first-video-demo/qa-12s-voiced/verified-snapshot-3s.mp4`: 3 seconds, 15 FPS, 512×640, 45 H.264 frames plus one AAC stream. The output refers to an actual ignored `verified-speech/voice.wav` rather than the source file.
- Representative real PNG frames visually inspected at 0, 1 and 2 seconds, contact sheet under `outputs/qa/verified-render-inputs/qa-frames.jpg`. Video codec/duration/stream checks passed, but this does **not** establish the quality of eyelid transitions or lip sync on other speakers.

## Limits and future review

Neither SHA-256 nor checked paths are a cryptographic trust boundary against **malicious concurrent same-user modification of the newly copied frame directory**. Node and FFmpeg do not hold immutable read-only file descriptors from validation through every subprocess read; a same-user attacker can intentionally mutate these writable render artifacts. For hostile multiuser execution, a separate OS sandbox with private immutable mounts, privileges and per-process filesystem isolation is needed. That is outside PuppetControl's present local developer scope.

Rig changes, repeated native speech samples, arbitrary-image eye deformation and cross-rig bridge animation still require independent perceptual QA. Keep stacked PRs in draft until reviewed.
15 changes: 9 additions & 6 deletions src/audio/cue-provenance.ts
Original file line number Diff line number Diff line change
Expand Up @@ -131,15 +131,18 @@ export async function createCueBinding(
throw new CueBindingError("CUE_BINDING_PATH_INVALID", "Cue binding sidecar must be a separate file");
}

const cueText = await readFile(cues.fullPath, "utf8");
const parsedCues = parseRhubarbMouthCuesJson(cueText);
// Parse the *same* bytes we hash. A separate filename reopen for the cue
// digest could otherwise hash a replacement file after parsing an older one.
const cueBytes = await readFile(cues.fullPath);
if (cueBytes.length < 1 || cueBytes.length > MAX_CUES_BYTES) {
throw new CueBindingError("CUE_BINDING_INPUT_INVALID", "Rhubarb cues are empty or oversized");
}
const parsedCues = parseRhubarbMouthCuesJson(cueBytes.toString("utf8"));
if (parsedCues.length === 0) {
throw new CueBindingError("CUE_BINDING_INPUT_INVALID", "Rhubarb cue file must contain at least one mouth cue");
}
const [audioSha256, cuesSha256] = await Promise.all([
fingerprint(audio.fullPath),
fingerprint(cues.fullPath),
]);
const cuesSha256 = createHash("sha256").update(cueBytes).digest("hex");
const audioSha256 = await fingerprint(audio.fullPath);
const binding = cueBindingSchema.parse({
format: CUE_BINDING_FORMAT,
version: CUE_BINDING_VERSION,
Expand Down
69 changes: 50 additions & 19 deletions src/render/rev2d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,18 @@ import { randomUUID } from "node:crypto";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { copyFile, link, lstat, mkdir, readFile, rename, stat, writeFile, unlink } from "node:fs/promises";
import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
import { basename, dirname, join, resolve } from "node:path";
import type { Character, Scene } from "../scene/schemas.js";
import { compilePerformance, PreviewUnsupportedError } from "./scene-plan.js";
import { parseRhubarbMouthCuesJson } from "../audio/rhubarb.js";
import { verifyCueBinding } from "../audio/cue-provenance.js";
import { snapshotBoundSpeech } from "./speech-snapshot.js";
import { buildFfmpegArgs, type FfmpegAudioInput } from "../audio/ffmpeg.js";
import type { VideoOutput } from "../video/output.js";
import type { SceneEvent } from "../scene/schemas.js";
import { planPoseTimeline, PoseTimelineError } from "../animation/pose-timeline.js";
import { verifyAvatarBuildRecord, checkAvatarPoseCoverage } from "../character/provenance.js";
import { checkedPath, pinnedRev2D } from "../character/avatar.js";
import { snapshotImportedRig } from "./rig-snapshot.js";

const execFileAsync = promisify(execFile);

Expand Down Expand Up @@ -40,6 +41,11 @@ interface Rev2DModel {
animations: unknown[];
}

/** Accept only Rev2D's flat PNG frame names, never paths from the subprocess. */
export function isSafeRev2DFrameName(value: unknown): value is string {
return typeof value === "string" && /^frame_[0-9]{4,8}\.png$/.test(value);
}

export interface FrameResult {
framesDir: string;
count: number;
Expand All @@ -54,13 +60,28 @@ export interface FrameResult {
camera?: { events: SceneEvent[]; sourceWidth: number; sourceHeight: number };
}

function insideRoot(path: string, root = process.cwd()): string {
const absolute = resolve(path);
const diff = relative(resolve(root), absolute);
if (diff === ".." || diff.startsWith(".." + sep) || isAbsolute(diff)) {
throw new PreviewUnsupportedError("PATH_OUTSIDE_WORKSPACE", "Path must be inside the project directory: " + absolute);
/**
* Frame directories are created once by this render, never reused.
* Resolve existing ancestors before mkdir so symlink/junction escapes are
* rejected even when the final target does not exist yet.
*/
export async function prepareFreshFrameDirectory(
outputDirectory: string, root = process.cwd(),
): Promise<string> {
const destination = await checkedPath(root, resolve(outputDirectory));
await mkdir(dirname(destination), { recursive: true });
await checkedPath(root, destination);
try {
await mkdir(destination);
} catch (error) {
if (error instanceof Error && "code" in error && error.code === "EEXIST") {
throw new PreviewUnsupportedError("FRAME_DIRECTORY_EXISTS",
"This render directory already exists; choose a fresh path to avoid overwriting frames");
}
throw error;
}
return absolute;
await checkedPath(root, destination);
return destination;
}

async function execute(binary: string, args: string[], cwd: string, timeout: number): Promise<string> {
Expand Down Expand Up @@ -105,6 +126,7 @@ async function readRig(modelPath: string, scene: Scene): Promise<Rev2DModel> {
async function loadSpeech(
scene: Scene,
sceneFilePath: string,
framesDir: string,
): Promise<{
speech?: { cues: ReturnType<typeof parseRhubarbMouthCuesJson>; startMs: number };
audio?: FfmpegAudioInput;
Expand All @@ -119,13 +141,18 @@ async function loadSpeech(
const cuesFile = await checkedPath(process.cwd(), resolve(directory, entry.cuesPath));

if (entry.bindingPath) {
const bindingFile = await checkedPath(process.cwd(), resolve(directory, entry.bindingPath));
const verified = await verifyCueBinding(source, cuesFile, bindingFile);
if (!verified.valid) {
throw new PreviewUnsupportedError(verified.code, verified.message);
}
const binding = await checkedPath(process.cwd(), resolve(directory, entry.bindingPath));
const verifiedSnapshot = await snapshotBoundSpeech(
source, cuesFile, binding, join(framesDir, "verified-speech"), process.cwd(),
);
return {
speech: { cues: verifiedSnapshot.cues, startMs: entry.startMs },
audio: { path: verifiedSnapshot.audioPath, startMs: entry.startMs },
};
}

// Legacy unbound scenes remain supported for compatibility. They do not
// claim protection against post-read changes to their source audio bytes.
try {
const file = await stat(source);
if (!file.isFile() || file.size < 1) throw new Error("Speech file must be a nonempty file");
Expand Down Expand Up @@ -218,7 +245,8 @@ async function renderRigSegment(input: {
args.push("--times", frameTimes.join(","), ...rev2dFrameRenderingFlags(scene));
const stdout = await execute(process.execPath, args, rev2dRoot, 240_000);
const response = JSON.parse(stdout) as { files?: string[]; width?: number; height?: number };
if (!Array.isArray(response.files) || response.files.length !== frameTimes.length) {
if (!Array.isArray(response.files) || response.files.length !== frameTimes.length ||
response.files.some((entry) => !isSafeRev2DFrameName(entry))) {
throw new PreviewUnsupportedError("RENDER_FRAME_COUNT_MISMATCH", "Rig segment rendered wrong number of frames.");
}
if (response.width !== scene.resolution.width || response.height !== scene.resolution.height) {
Expand Down Expand Up @@ -294,11 +322,13 @@ export async function renderRev2DFrames(
}

const rev2dRoot = await checkedPath(process.cwd(), resolve(process.env.PUPPETCONTROL_REV2D_ROOT ?? ".cache/rev2d"));
const framesDir = insideRoot(outputDirectory);
const { speech, audio } = await loadSpeech(scene, sceneFilePath);
const familyNames: string[] = [];

await mkdir(framesDir, { recursive: true });
const framesDir = await prepareFreshFrameDirectory(outputDirectory);
const { speech, audio } = await loadSpeech(scene, sceneFilePath, framesDir);
const isolatedRig = character.provenance ? await snapshotImportedRig(
characterPath, character.render.modelPath,
join(framesDir, "verified-actor-assets"), process.cwd(),
) : undefined;
let totalCopied = 0;
for (const [segmentIndex, segment] of plan.segments.entries()) {
// An interpolated transition occupies real output frames and uses the same
Expand All @@ -311,7 +341,8 @@ export async function renderRev2DFrames(
// Different artwork must use separately declared rig assets; sharing rigId
// alone does not create a visual pose change.
const rigFile = family.modelPath ?? character.render.modelPath;
const modelPath = await checkedPath(process.cwd(), resolve(dirname(characterPath), rigFile));
const modelPath = isolatedRig?.modelPath ??
await checkedPath(process.cwd(), resolve(dirname(characterPath), rigFile));
const model = await readRig(modelPath, scene);
const segmentDir = join(framesDir, "segment_" + String(segmentIndex).padStart(3, "0"));
const files = await renderRigSegment({
Expand Down
Loading
Loading