Repository navigation
Synthesise a soundtrack on the same clock as the cuts #331
Description
Part of #326. Batch 2 — depends on #336 for TimePoint and the beat grid.
AudioTrack requires a src. An LLM can write a score and cannot hand over a WAV, so every generated scenario is silent. Both reference reels synthesise their entire soundtrack from oscillators, noise and filters, with no audio file anywhere, and the SVG one renders it offline into a buffer before muxing. This is the single largest gap between a generated scenario and a generated reel, measured by how much of the perceived result it carries.
Owned files
crates/rustmotion-core/src/audio/(new:voices.rs,score.rs,synth.rs,dsp.rs,mod.rs)crates/rustmotion/src/encode/audio/(the offline render and the mux path)
Do NOT write: schema/scenario.rs (#336 registers the audio.synth field), anything under engine/.
Deliverables
-
A declarative score sharing the visuals' clock:
"audio": { "bpm": 115, "voices": { "kick": { "type": "sine", "freq": [150, 42], "sweep": 0.14, "decay": 0.38 }, "hat": { "type": "noise", "filter": {"type":"highpass","freq":7500}, "decay": 0.05 } }, "score": [ { "voice": "kick", "every": "1b", "from": "@2.2s", "to": "@12.6s" }, { "voice": "hat", "every": "1b", "offset": "0.5b", "from": "@2.2s" } ], "master": { "compressor": { "threshold": -14, "ratio": 4 }, "limiter": true } }
every,from,atandoffsetgo through Place scenes on an absolute timeline with a beat grid #336'sTimePointparser. Sharingbpmandbeat_offsetwith the cuts is the whole point: it is what makes an edit land on a downbeat rather than near one. -
A small, boring synth. Sine, square, saw and triangle oscillators; a noise buffer; biquad filters in lowpass, highpass, bandpass and notch; ADSR envelopes; a bus with gain, a compressor and a limiter. Rendered offline to an f32 buffer at 48 kHz.
-
Coexistence with file-based tracks.
audioaccepts both the existingAudioTrackentries and a synth block in the same scenario, mixed into one bus. -
The limiter is a default, not an option. The SVG reel's author found his first mix clipping at 0 dB only by reading
ffmpegoutput after the fact. -
--frames a-bkeeps its audio slice. A segment renders its own slice of the synthesised buffer, not a restart from zero, matching what the existing file-based path already guarantees.
Out of scope for this workstream: convolution reverb, sample loading, any effects rack. A synth that grows is a synth that stops being reviewable.
Acceptance
- A scenario with no audio file and a 60-event score produces a muxed soundtrack
- Two renders of that scenario produce byte-identical audio
- Peak never exceeds -0.1 dBTP with the limiter on
Activity
[-]WS-5 motion: repeat, yoyo, steps easing and a declarative camera shake[/-][+]WS-6 audio: a synthesised score on the visuals' clock[/+][-]WS-6 audio: a synthesised score on the visuals' clock[/-][+]Synthesise a soundtrack on the same clock as the cuts[/+]LeadcodeDev commentedon Sep 26, 2026
Delivered on
mainat 187cf7c (squash of #340). Verified present onmain, not taken on the PR's word.fix(still, sheet): analyse the audio track before exporting a frame (#…