diff --git a/quest/m1/README.md b/quest/m1/README.md index c0533902579..4a4453d09c4 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -142,7 +142,7 @@ above them. - [Watch worker](/quest/m1/watch-worker.md) - watch playback runs in a worker onto an OffscreenCanvas, so main-thread jank never stalls video or audio - [Rust papercuts](/quest/m1/papercuts-rs.md) - HTTP refusals are counted, and a `u64::MAX` resume is unbounded - [Duration parser defaults](/quest/m1/cli-duration-defaults.md) - a moq-tokio duration set in code beats the parser default on a parsed or merged config, zero included, after #5350 -- [Publish channel count](/quest/m1/publish-audio-channel-count.md) - forcing a channel count on an Audio.Capture stops costing the subscriber gaps of silence +- [Publish channel count](/quest/m1/publish-audio-channel-count.md) - forcing a channel count on an Audio.Capture stops costing the subscriber gaps of silence, and the media fixture's cold-start audio-tone flake is fixed at its cause - [JS request deadline](/quest/m1/js-request-deadline.md) - each JS request, PUBLISH_NAMESPACE included, has one fatal 10 s timer from create to answer and fails fast without stream credit, so nothing queues or retries - [Handshake deadline covers admission](/quest/m1/handshake-admission-deadline.md) - `listen.timeout` bounds a connection through auth admission on both listeners, replacing the hardcoded admission timeout - [Container reads drop backfill](/quest/m1/read-backfill.md) - the JS container consumer drops groups below its read cursor like Rust diff --git a/quest/m1/publish-audio-channel-count.md b/quest/m1/publish-audio-channel-count.md index dddfafba752..154758631bd 100644 --- a/quest/m1/publish-audio-channel-count.md +++ b/quest/m1/publish-audio-channel-count.md @@ -2,63 +2,53 @@ ## Goal -`new Audio.Capture({ source, channelCount: 1 })` publishes continuous audio. -A subscriber hears no gaps that the same capture without the override does not -also have, so the documented way to pin a publisher's channel count stops being -a way to degrade it. +`new Audio.Capture({ source, channelCount: 1 })` publishes audio a subscriber +plays as smoothly as the same capture without the override, so pinning a +publisher's channel count stops being a way to degrade it. The browser media +fixture can then pin mono again. -Boundaries: `channelCount` keeps its meaning, "force the captured channel -count", and the published shape does not move. Whether the fix is to stop -forcing `channelCountMode: "explicit"`, to do the downmix off the realtime -graph, or to keep the mode and remove whatever it starves, is the quest's to -decide. +This is also the home for the media fixture's cold-start audio-tone flake +without the pin, which shows the same symptom. Fix it at its cause, not by +lowering the tone floor. -## Plan - -`js/publish/src/audio/capture.ts` sets -`channelCountMode: requestedChannels !== undefined ? "explicit" : "max"`, so -supplying `channelCount` is what puts the AudioWorklet behind a forced Web -Audio downmix. Omitting it does not always take the `"max"` path: -`requestedChannels` falls back to `requestedChannelCount(source.track)`, -so an applied `getUserMedia` channelCount constraint forces `"explicit"` too. -Account for that path when isolating the cause. - -What was measured, on macOS headless Chromium through `just test media`, -publishing a `MediaStreamAudioDestinationNode` track into a graph already -running at 48 kHz: - -- With `sampleRate: 48000, channelCount: 1`: 11 probe samples across a run read - a flat spectrum (peak equal to median, below `AUDIBLE_RMS`), which is - silence, not a weak tone. The `audio tone` check fell to 82-88% against its - 90% floor and took two negative controls down with it. -- With both overrides removed: zero such samples, and every media check passed. -- `main`, whose fixture pinned the same format on the encoder rather than the - capture, reported `tone 150dB above the floor` and no silent samples at all. +`channelCount` keeps its meaning, "force the captured channel count". -What is not established, and should be first: - -- The two overrides were removed together, so `channelCount` is implicated by - reading the `channelCountMode` choice, not by an isolated run. Re-run with - `sampleRate` alone and with `channelCount` alone before believing the - attribution. -- The mechanism. A forced downmix starving or stalling the worklet is the - obvious guess and is not evidence. -- Whether a `getUserMedia` microphone track shows it, or only a - destination-node track, whose channel count Web Audio reports as 2 by - default. `requestedChannelCount` exists for the macOS - mono-mic misreport, so that path has a real caller and cannot simply lose the - override. - -The regression belongs where the evidence came from: the browser fixture can -pin a channel count again once this is fixed, and -`test/interop/clients/js/src/fixture.ts` carries a comment saying why it does -not. A unit test that counts worklet callbacks under an explicit downmix would -be cheaper than a full media run, if one can be made to fail reliably. +## Plan -The `just test media` audio-tone misses under load were not this: the fixture -takes the `"max"` path, and every miss was the player's own buffering silence -with audio still arriving. +What is known: + +- The real-worklet matrix in + `test/interop/clients/js/capture-channel-count.browser.ts` (in + `just test harness`) finds nothing override-specific: no timestamp gaps, + the requested format, and audible output in every channel, on Linux and + macOS. It only asserts each channel is audible at least once. An uncommitted + per-quantum trace (#5270) found one short silent run during startup in + every case, the no-override baseline included, and none after. Under heavy + load the baseline also showed a later run, so a strict per-quantum check + flakes on the source. Tightening the matrix needs a baseline-relative + measure. +- The player stalls. In CI (#5270), the pinned fixture failed the `just test media` + audio-tone check in 2 of 3 runs, against ~5% of unpinned runs (5 of 108, + all cold start). Failures look the same either way: player samples go + `stalled` while received audio bytes keep growing, and the jitter delay jumps + from ~77 ms to 500-700 ms. + +The working hypothesis is that the explicit downmix (`channelCountMode: +"explicit"` in `js/publish/src/audio/capture.ts`) changes something the player +reacts to, such as frame cadence or burstiness, rather than dropping PCM. +Neither half is proven. Measure the +publisher's frame arrival timing pinned vs unpinned before picking a fix, and +check whether the player's re-stall behavior +([watch jitter target](/quest/m1/audio-jitter-target/watch.md)) is the +amplifier. A loaded local host fails the media fixture either way, so compare +on CI or a quiet machine. + +Not covered: an applied `getUserMedia` channelCount constraint also takes the +explicit path, but a destination-node track refuses that constraint, so the +harness cannot exercise it. ## Related +- [Watch jitter target](/quest/m1/audio-jitter-target/watch.md) - a ring that runs dry re-stalls and refills to the target +- [Test flakes, round 2](/quest/m1/test-flakes-2/README.md) - load flakes fixed at their cause - [Browser benchmarks](/quest/m2/browser-benchmarks.md) - the other place browser-side capture and encode costs get measured diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index a260c2ec95e..f4b81052287 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -45,3 +45,4 @@ Public API: none. Wire: none. ## Related - [Controlled time](/quest/m1/time/README.md) - the systemic fix these rules ask for: every crate on a controlled clock +- [Publish channel count](/quest/m1/publish-audio-channel-count.md) - tracks the media fixture's cold-start audio-tone flake, a player stall it shares with the mono pin diff --git a/test/interop/clients/js/capture-channel-count.browser.ts b/test/interop/clients/js/capture-channel-count.browser.ts new file mode 100644 index 00000000000..9f248210f99 --- /dev/null +++ b/test/interop/clients/js/capture-channel-count.browser.ts @@ -0,0 +1,25 @@ +/** Capture diagnostic uses the real browser and worklet, with each format override isolated. */ +import { expect, test } from "bun:test"; +import { createServer } from "vite"; +import { launch } from "./harness"; + +test("capture format override matrix", async () => { + const server = await createServer({ server: { host: "127.0.0.1", port: 0 } }); + await server.listen(); + const browser = await launch(["--autoplay-policy=no-user-gesture-required"]); + try { + const page = await browser.newPage(); + await page.goto(`${server.resolvedUrls?.local[0]}capture-channel-count.html`); + await page.waitForFunction(() => typeof window.captureMatrix === "function"); + const result = await page.evaluate(() => window.captureMatrix()); + expect(result).toEqual([ + { mode: "none", channelsAudible: [true, true], gaps: 0, format: { sampleRate: 48_000, channelCount: 2 } }, + { mode: "rate", channelsAudible: [true, true], gaps: 0, format: { sampleRate: 48_000, channelCount: 2 } }, + { mode: "channels", channelsAudible: [true], gaps: 0, format: { sampleRate: 48_000, channelCount: 1 } }, + { mode: "both", channelsAudible: [true], gaps: 0, format: { sampleRate: 48_000, channelCount: 1 } }, + ]); + } finally { + await browser.close(); + await server.close(); + } +}, 30_000); diff --git a/test/interop/clients/js/capture-channel-count.html b/test/interop/clients/js/capture-channel-count.html new file mode 100644 index 00000000000..d19426181b3 --- /dev/null +++ b/test/interop/clients/js/capture-channel-count.html @@ -0,0 +1,2 @@ + + diff --git a/test/interop/clients/js/harness.browser.ts b/test/interop/clients/js/harness.browser.ts index 676add78a8a..ebe1d35c81e 100644 --- a/test/interop/clients/js/harness.browser.ts +++ b/test/interop/clients/js/harness.browser.ts @@ -22,7 +22,7 @@ test("pause activates its button when a canvas intercepts pointer input", async } finally { await browser.close(); } -}); +}, 30_000); for (const stall of ["audio", "video"]) { test(`the paused play button stays clickable during ${stall} stalling`, async () => { diff --git a/test/interop/clients/js/src/capture-channel-count.ts b/test/interop/clients/js/src/capture-channel-count.ts new file mode 100644 index 00000000000..f4d6701edbc --- /dev/null +++ b/test/interop/clients/js/src/capture-channel-count.ts @@ -0,0 +1,61 @@ +/** Real capture measurements isolating format overrides before encoding or playout. */ +import * as Publish from "@moq/publish"; +import { Effect } from "@moq/signals"; + +declare global { + interface Window { + captureMatrix: () => Promise; + } +} + +window.captureMatrix = async () => { + const results = []; + for (const mode of ["none", "rate", "channels", "both"]) { + const source = new AudioContext({ sampleRate: 48_000 }); + const oscillator = source.createOscillator(); + const destination = source.createMediaStreamDestination(); + oscillator.connect(destination); + oscillator.start(); + await source.resume(); + const track = destination.stream.getAudioTracks()[0]; + const capture = new Publish.Audio.Capture({ + source: { track: track as Publish.Audio.StreamTrack, kind: "music" }, + sampleRate: mode === "rate" || mode === "both" ? 48_000 : undefined, + channelCount: mode === "channels" || mode === "both" ? 1 : undefined, + }); + const effect = new Effect(); + try { + let fanout = capture.out.frames.peek(); + while (!fanout) { + await capture.out.frames.changed(); + fanout = capture.out.frames.peek(); + } + const reader = fanout.subscribe(effect).getReader(); + const channelsAudible: boolean[] = []; + let gaps = 0; + let expected: number | undefined; + for (let i = 0; i < 512; i++) { + const { value: frame } = await reader.read(); + if (!frame) throw new Error("capture ended"); + for (const [channel, samples] of frame.channels.entries()) { + if (!samples.length || samples.some((sample) => !Number.isFinite(sample))) { + throw new Error(`capture channel ${channel} has invalid PCM`); + } + const rms = Math.sqrt(samples.reduce((sum, value) => sum + value * value, 0) / samples.length); + channelsAudible[channel] = (channelsAudible[channel] ?? false) || rms > 0.01; + } + if (expected !== undefined && Math.abs(frame.timestamp - expected) > 2) gaps++; + expected = frame.timestamp + (frame.channels[0].length / 48_000) * 1_000_000; + } + await reader.cancel(); + results.push({ mode, channelsAudible, gaps, format: capture.out.format.peek() }); + } finally { + effect.close(); + capture.close(); + track.stop(); + oscillator.stop(); + await source.close(); + } + } + return results; +}; diff --git a/test/interop/clients/js/src/fixture.ts b/test/interop/clients/js/src/fixture.ts index dbbe25ab678..732754be118 100644 --- a/test/interop/clients/js/src/fixture.ts +++ b/test/interop/clients/js/src/fixture.ts @@ -109,10 +109,10 @@ export class Fixture { // suspended graph. // // No sampleRate or channelCount override: the graph already runs at SAMPLE_RATE, and asking - // the capture for one channel puts the AudioWorklet behind an explicit Web Audio downmix - // whose output drops PCM under load, which reaches the player as gaps of silence. Publishing - // the destination node's own format costs nothing here, since the tone is the same in both - // channels. + // the capture for one channel puts the AudioWorklet behind an explicit Web Audio downmix that + // made the player stall far more often in CI, with no capture loss found in isolation; see + // /quest/m1/publish-audio-channel-count.md. Publishing the destination node's own format + // costs nothing here, since the tone is the same in both channels. const audioCapture = new Publish.Audio.Capture({ source: { track: audioTrack, kind: "music" }, }); diff --git a/test/interop/clients/js/tsconfig.json b/test/interop/clients/js/tsconfig.json index 45fd6397c6f..884d98a8dfe 100644 --- a/test/interop/clients/js/tsconfig.json +++ b/test/interop/clients/js/tsconfig.json @@ -8,6 +8,7 @@ "driver.ts", "harness.ts", "harness.browser.ts", + "capture-channel-count.browser.ts", "buffering", "media.ts", "src", diff --git a/test/justfile b/test/justfile index f01678bd6ac..ca585fb6ae3 100644 --- a/test/justfile +++ b/test/justfile @@ -26,7 +26,7 @@ harness: cd .. && bun test test/interop/compat/resolve.test.ts cd .. && bun install --frozen-lockfile cd interop/clients/js && bunx playwright install chromium - cd interop/clients/js && bun test ./harness.browser.ts ./src/contract.test.ts ./src/resume.test.ts + cd interop/clients/js && bun test ./harness.browser.ts ./capture-channel-count.browser.ts ./src/contract.test.ts ./src/resume.test.ts # Cross-language media interop test, built from this checkout. Stands up a # relay and runs the publisher x subscriber matrix. Default: rust only (fast