diff --git a/quest/m1/audio-jitter-target/README.md b/quest/m1/audio-jitter-target/README.md index fa2f3fae71b..882f7f136c7 100644 --- a/quest/m1/audio-jitter-target/README.md +++ b/quest/m1/audio-jitter-target/README.md @@ -48,6 +48,7 @@ native playback against its budgets. - [Watch](/quest/m1/audio-jitter-target/watch.md) - the browser plays the measured real-microphone rows with zero steady-state underruns by time-stretching the ring - [Native stretch](/quest/m1/audio-jitter-target/native-stretch.md) - `moq play` copies NetEq's accelerate and preemptive expand, sharing the browser's algorithm and vectors, so its native rows reach zero steady-state underruns +- [Buffered converge](/quest/m1/audio-jitter-target/buffered-converge.md) - a buffered `` time-stretches its audio back onto the shared clock after a stall or drift, so lip sync returns and no audio group is dropped - [Capped budgets](/quest/m1/audio-jitter-target/capped-budgets.md) - the AAC browser budgets are re-measured from nightlies after the TS importer groups each PES ## Closes diff --git a/quest/m1/audio-jitter-target/buffered-converge.md b/quest/m1/audio-jitter-target/buffered-converge.md new file mode 100644 index 00000000000..e1878c4de3d --- /dev/null +++ b/quest/m1/audio-jitter-target/buffered-converge.md @@ -0,0 +1,72 @@ +# [M] Buffered audio converges on the shared clock + +## Goal + +A `` with a nonzero `buffer` keeps lip sync in steady state: once +a stall, a rise of the `"auto"` delay, or publisher clock drift has left the +audio playhead behind the shared clock, the ring time-stretches back onto it. +It never skips, so a TTS utterance still plays every word. Today buffered +mode turns off the ring's catch-up (`playing = !this.#stalled && +!this.#buffered` in `js/watch/src/audio/ring-buffer.ts`), so any lag is +permanent. Audio trails video for the rest of the stream, and once the lag +passes `delay + buffer` the container consumer drops whole audio groups +("skipping slow groups"), which is the cut-out reported in +[moq-dev/moq#8031](https://github.com/moq-dev/moq/issues/8031). + +Boundaries: video already follows the shared clock and is unchanged. The +clock is not driven by audio, and a slow audio track never holds video. Live +mode (`buffer` of zero) keeps its own catch-up. Native playback has no +buffered mode. + +## Plan + +Planned 2026-11-06 (maintainer, /quest-import of +[moq-dev/moq#8031](https://github.com/moq-dev/moq/issues/8031)). A +simulation on main with the real `Sync`, container `Consumer`, `Backpressure` +and `AudioRingBuffer` (a copy of `js/watch/src/audio/replay.ts` with buffered +mode on) measured audio behind video: about -12 ms live versus +105 ms +buffered under `"auto"` with jittery then calm phases; 0 versus +300 ms after +400 ms stalls with `delay=100ms buffer=500ms`; and a publisher clock 1% fast +climbing to 1.4 s of lag with 2926 slow-group skips. The reporter's other +claim, that a delay above 500 ms breaks lip sync, did not reproduce (300, 800 +and 1500 ms all held about -22 ms). + +Decisions: + +- Direction: ✅ converge buffered audio by time-stretching / won't fix, + reply and close / sharpen docs and warn only. Steady-state lip sync comes + from each track converging on the shared clock, and buffered mode broke + that even for TTS with video. +- Rename `buffer`: keep / ✅ rename in m2, as + [Rename buffer to lookahead](/quest/m2/watch-lookahead.md). +- Milestone: ✅ m1, in this epic / m3. + +Recommendations: + +- Reuse [Watch](/quest/m1/audio-jitter-target/watch.md)'s stretch rather + than a second mechanism: buffered mode measures the playhead's lag behind + the shared clock and accelerates until it is back within the floor. It + never stretches ahead of the clock, since future-dated frames are + buffered mode's purpose. +- Extend the replay harness with buffered rows (the stall and the drift + scenarios above), budgeting audio-behind-video and slow-group skips, and + keep a TTS row asserting no frame is dropped. +- Update the `buffer` docs in `js/watch/src/element.ts`, `js/watch/src/sync.ts` + and `doc/lib/js/watch.md` to say live streams take their slack from + `delay`, and that a buffered player catches up after a stall by playing + slightly faster. + +Public API: none expected. Wire: none. + +## Required + +- [Watch](/quest/m1/audio-jitter-target/watch.md) - lands the ring's time stretch this reuses + +## Closes + +- [#8031](https://github.com/moq-dev/moq/issues/8031) - buffered playback loses lip sync and cuts audio out on a live stream + +## Related + +- [Rename buffer to lookahead](/quest/m2/watch-lookahead.md) - the name that invited the misuse +- [Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md) - the other buffered-mode change to the audio path diff --git a/quest/m2/README.md b/quest/m2/README.md index 88d6f5f7f1c..a04de05388e 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -184,6 +184,7 @@ quests that require them. - [Session policy lives in moq-net](/quest/m1/session-policy/README.md) - accept and connect deadlines, admission, window widening and reconnect live once in moq-net; moq-tokio and moq-relay's io_uring path call them (held in m1 by an open claim) - [Rename moq-rtc to moq-webrtc](/quest/m2/webrtc-rename.md) - one published `moq-webrtc` crate holds the gateway, the ICE driver, and the data-channel transport - [MoQ on rind](/quest/m1/rind/README.md) - every MoQ crate takes its crypto from rind, a pure-Rust graviola fork, so no build compiles C crypto (held in m1 by an open claim) +- [Rename buffer to lookahead](/quest/m2/watch-lookahead.md) - ``'s `buffer` becomes `lookahead`, so it is no longer mistaken for a jitter buffer on live streams - [Cut readiness](/quest/m2/cut-gate/README.md) - the checks the cut needs that break nothing: main no slower than release, vendored fixes ported, the pre-cut audit re-run, and moq-c at parity - [Probe](/quest/m2/probe/README.md) - a session probes up to the bitrate it wants to send or receive, so a stopped simulcast rendition resumes and a viewer up-switches; the browser pads where native sessions retransmit (held in m2 by an open claim; moves to m3) - [Finish the milestone re-cut](/quest/m2/recut-held.md) - claim-held epics move to the milestone their kind and priority give them once their claims merge diff --git a/quest/m2/watch-lookahead.md b/quest/m2/watch-lookahead.md new file mode 100644 index 00000000000..231daf1a020 --- /dev/null +++ b/quest/m2/watch-lookahead.md @@ -0,0 +1,41 @@ +# [S] Rename moq-watch's buffer to lookahead + +## Goal + +``'s `buffer` property and attribute, and the `Player`/`Sync` +input behind them, are named `lookahead`. Users read `buffer` as a jitter +buffer and set it on live streams, where it disables catch-up +([moq-dev/moq#8031](https://github.com/moq-dev/moq/issues/8031)). It is +really how far future-dated media, such as TTS written faster than real time, +may run ahead of the live edge. Live slack stays `delay`. + +Boundaries: JavaScript only. Nothing in Rust, Python, or the bindings uses +the name. The buffered-ranges UI (`buffer-control.ts`) shows something else +and keeps its name. + +## Plan + +Planned 2026-11-06 (maintainer, /quest-import of +[moq-dev/moq#8031](https://github.com/moq-dev/moq/issues/8031)). + +Decisions: + +- Rename: keep `buffer` / ✅ rename in m2. Breaking for `@moq/watch`, so it + joins the next breaking release. +- Name: ✅ `lookahead` / `ahead` / `burst`. It says what the value is, and + is already the word the docs use. + +Settled by repository rules: no deprecated alias. `buffer` is removed +outright, and `js/watch/MIGRATE.md` gets an Unreleased entry showing the move. +Internal names derived from it (`buffered`, `parseBuffer`) follow when they +mean the same thing. Update `doc/lib/js/watch.md`, +`doc/concept/use-case/ai.md`, the demos, and quests that name the property, +such as [Buffered playback subscribes in ascending +order](/quest/m1/buffered-ascending.md). + +Public API: breaking, `buffer` becomes `lookahead` on `` and the +`Player`/`Sync` inputs. Wire: none. + +## Related + +- [Buffered audio converges on the shared clock](/quest/m1/audio-jitter-target/buffered-converge.md) - the behavior fix for the same report