Skip to content

quest: drop audio-capture-lag, fixed on main by removing Opus DTX - #6599

Closed
kixelated wants to merge 4 commits into
mainfrom
quest/m1/audio-capture-lag
Closed

kixelated wants to merge 4 commits into
mainfrom
quest/m1/audio-capture-lag

Conversation

@kixelated

@kixelated kixelated commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Builds on #6595 (this branch was cut from it) until that merges; only the last commit belongs to this PR.

Deletes quest/m1/audio-capture-lag.md and its quest/m1/README.md entry. No code change: the measured cause is already fixed on main.

Measurement

Physical camera and microphone (AT2020 USB), Chrome, once a second: the worklet's latest stamp minus now(), the AudioContext clock minus the wall clock, empty and dropped quanta, encodeQueueSize, and encoder input and output counts.

Local js/publish build of this branch, camera (H.264 720p30) plus Opus, about 130 s: the stamp gap stayed flat (about +55 ms), no empty or dropped quanta, encodeQueueSize 0, every input produced an output, and the viewer's audio catalog delay was 13 ms. No drift.

Release moq.dev/publish (6f204fd), same devices, with AudioWorkletNode and AudioEncoder wrapped in the page:

t (s) frames in frames out input ts minus output ts input ts minus clock
174.7 1123 1030 1860 ms -32 ms
189.7 1874 1651 4460 ms -19 ms
202.2 2498 2029 9380 ms -29 ms

Neither quest hypothesis held. Input stamps track the clock, so no samples are lost, and encodeQueueSize stays 0, so there is no backlog. Release configures voice with usedtx: true, and Chromium stamps Opus output by counting the samples it emits, so each frame DTX suppresses pulls later audio 20 ms earlier: (in - out) × 20 ms equals the gap exactly. DTX's keep-alive every 400 ms matches the 400 ms catalog steps seen earlier.

main fixed this in #4808 (DTX off for voice) and #4919 (DTX removed, with a test that it never reaches the encoder). The backport to release is #6598.

API and wire impact

None.

Checks

just check and quest check pass.

Suggested follow-ups

  • The AudioContext clock, and so the worklet stamp, stepped behind the wall clock: about -130 ms in the first 70 s and then flat on the local run, and about -1.3 ms/s over 80 s on the release run. That is too short a sample to call it drift. An hour-long run would show whether it keeps accumulating.
  • demo/relay/localhost.toml fails at startup on this branch with stats.interval must be non-zero ([stats] enabled = true, no interval), so just dev cannot start the relay.

Decisions needed

quest/m1/audio-capture-lag.md exists only on #6595, which adds it (renamed from audio-capture-clock.md); it is not on main. So this PR has no diff of its own against main, and merging it now would also land all of #6595.

  1. Fold the deletion into feat(catalog)!: measure jitter and delay from timestamps, not the wall clock #6595 so it never adds the quest, move this measurement into its body, and close this PR as redundant. (recommended: one PR, and feat(catalog)!: measure jitter and delay from timestamps, not the wall clock #6595's body stops saying the camera lag is still open)
  2. Keep this a draft until feat(catalog)!: measure jitter and delay from timestamps, not the wall clock #6595 merges, then merge main in and mark it ready; the diff is then just the deletion.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 4 commits October 23, 2026 16:03
…l clock

The catalog estimator measured each frame's flush time against its media
timestamp, so any capture clock drifting from the wall clock read as
ever-rising lateness. A physical camera on moq.dev/publish ratcheted audio
jitter and delay without bound, republishing the catalog and driving
hundreds of SUBSCRIBE_UPDATEs a minute.

jitter now covers only reorder delay and batch spans. delay is how far a
frame's timestamp trails the highest one another track of the catalog
already wrote, so importers measure it too. flush, moq_publish_media_flush,
the binding wrappers, and moqsink's encoder pad property are removed.

The camera's audio still drifts behind video by PTS; the subscribe update
quest becomes the audio capture clock fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on a physical camera and microphone. Neither quest hypothesis
held: worklet stamps track the wall clock and encodeQueueSize stays 0.
Release enables Opus DTX for voice, and Chromium stamps Opus output by
counting emitted samples, so each suppressed silent frame pulls later
audio 20 ms earlier. main already removed DTX (#4808, #4919); a local
main build shows no drift.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: measured on a physical camera and microphone. The audio lag is Opus DTX on release (Chromium stamps Opus output by emitted samples, so each suppressed silent frame pulls later audio 20 ms earlier). It is neither of the quest's two hypotheses. main already fixed it in #4808 and #4919, so this PR only deletes the quest. The backport to release is #6598. just check passed. Ready for review once #6595 merges.

(Written by Claude Opus 5.5)

kixelated added a commit that referenced this pull request Oct 24, 2026
Folds in #6599's measurement: release's DTX made Chromium's
sample-counted Opus stamps fall behind, and main already removed DTX.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Closing: this PR's quest deletion and its Opus DTX measurement were folded into #6595 (credited there), which has merged. Thanks for the measurement.

(Written by Claude Opus 5.5)

@kixelated kixelated closed this Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant