From e65f06bdb820c986a5a6dd67e51d528401539f78 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 15 Oct 2026 08:51:20 -0700 Subject: [PATCH 1/2] quest: import #5521, #5563, #5564 and #5565 - m0: a debug moq-relay starts on Windows (#5563) - m1: backport the Lite SETUP violation closes to release (#5521) - m1: a TS export never exits on start-up latency (#5564) - fold #5565's evidence into the TS decoded-content flake quest Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 1 + quest/m0/relay-windows-stack.md | 49 +++++++++++++++++++ quest/m1/README.md | 2 + quest/m1/lite-setup-backport.md | 46 ++++++++++++++++++ quest/m1/test-flakes-2/ts-video-content.md | 24 ++++++++++ quest/m1/ts-drift-startup.md | 56 ++++++++++++++++++++++ 6 files changed, 178 insertions(+) create mode 100644 quest/m0/relay-windows-stack.md create mode 100644 quest/m1/lite-setup-backport.md create mode 100644 quest/m1/ts-drift-startup.md diff --git a/quest/m0/README.md b/quest/m0/README.md index 0d3a369914..ad62d65b2e 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -81,6 +81,7 @@ have fixes on `main`: macOS tests (#5335), drill-sensitivity (#5338), loom - [Draft-22 media on 0.17](/quest/m0/release-22/README.md) - a 0.17.x with the LOCATION_FILTER and FIRST_OBJECT fixes and moq-noq 1.3.5, then a patch with draft-20+ FETCH and the unset-clock and quiet-edge fixes - [nixpkgs ships the lld fix](/quest/m0/nixpkgs-lld-fix.md) - waiting on NixOS/nixpkgs#568680 to reach `nixpkgs-unstable`, which the release build fix bumps to - [Release builds link](/quest/m0/release-link.md) - the Linux zigbuild release binaries and moq-gst packages link again after the 2026-10-11 nixpkgs bump, so the next release cut can publish +- [Windows relay stack](/quest/m0/relay-windows-stack.md) - a debug moq-relay starts on Windows instead of overflowing its 1 MiB main-thread stack (#5563) - [web-transport releases the qmux fixes](/quest/m0/qmux-credit-upstream.md) - waiting on moq-dev/web-transport#412 and #413 to merge and ship, which qmux credit bumps to - [qmux credit](/quest/m0/qmux-credit.md) - qmux returns connection credit for dropped and stopped streams and delivers its close frame, on both lines - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh route epoch, the newest wins a path, and only routes with the same epoch resume a subscription diff --git a/quest/m0/relay-windows-stack.md b/quest/m0/relay-windows-stack.md new file mode 100644 index 0000000000..bc2e7717ed --- /dev/null +++ b/quest/m0/relay-windows-stack.md @@ -0,0 +1,49 @@ +# [S] A debug moq-relay starts on Windows + +## Goal + +A debug (`dev` profile) `moq-relay` starts on Windows, where the main thread +has a 1 MiB stack, instead of dying with `STATUS_STACK_OVERFLOW` before it +logs anything. Embedders that call `Relay::load(..).await` and `.run().await` +from their own Windows main thread are covered too, not only the binary. + +## Plan + +Reported in [#5563](https://github.com/moq-dev/moq/issues/5563): `cargo run +--bin moq-relay -- prod.toml` in PowerShell or the MinGW console overflows the +main thread, with no output even at `RUST_LOG=trace`. It works under WSL, +which has an 8 MiB main thread. + +Likely cause, not yet confirmed: `rs/moq-relay/src/main.rs` is one +`#[tokio::main]` block that inlines `Relay::load`'s future, the `Relay`, and +`Relay::run`'s future into a single unboxed state machine. `block_on` builds +and polls that state machine on the main thread, and debug builds copy it by +value several times. Logging starts inside `Config::load`, which is why +nothing prints. The relay tests already sidestep this with 32 MiB threads +(`rs/moq-relay/tests/goaway_cluster.rs` and others), and +`rs/moq-relay/src/connection.rs` and `rs/moq-tokio/src/client.rs` already box +large futures for the same reason. + +Decided 2026-10-15 (maintainer): + +- Fix at the source by boxing the large futures inside the library: + `Relay::load` and `Relay::run` box their large inner futures, following the + precedent in `connection.rs` and `client.rs`. `main` stays thin. Neither a + bigger thread in `main` nor a `/STACK` linker flag; both only add headroom + for the binary. +- Confirm first: measure `size_of_val` of the main future, `Relay::load(..)` + and `run()` in a debug build, and box what is actually large. +- Regression test: drive the relay's load and run with a minimal config on a + thread with a 1 MiB stack in a debug build. It must overflow without the fix + on any OS, so Linux CI catches it too. +- Windows CI adds `-p moq-relay` to `windows-test` (`rs/justfile`), so the + relay's tests, the regression above included, run on Windows. + +No release backport is planned; release builds have smaller futures. If the +fix shows `release` overflows too, ask the maintainer. + +Public API: none expected (boxing is internal). Wire: none. + +## Closes + +- [#5563](https://github.com/moq-dev/moq/issues/5563) - Relay: Stack overflow error launching relay on Windows diff --git a/quest/m1/README.md b/quest/m1/README.md index 2acee7fc5c..bcf576eddb 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -74,6 +74,7 @@ above them. - [FETCH max-delay](/quest/m1/fetch-max-delay.md) - historical readers obey publisher max-age and an optional tighter reader budget, including while holding a group - [Draft-20 FETCH edge-case backport](/quest/m1/fetch-edge-cases.md) - backport the final-object and response-group fixes to a later 0.17 patch after main lands - [Subscribe-close backport](/quest/m1/subscribe-close-backport.md) - the same fix and the `race` loser fix on `release`, so closing a subscription during setup leaves no unhandled rejection in a later 0.17 patch +- [Lite SETUP backport](/quest/m1/lite-setup-backport.md) - `release` closes a moq-lite session on a repeated SETUP parameter or a second Setup Stream, in a later 0.14.x patch (#5521) - [Claim answer epochs](/quest/m1/claim-answer-epochs.md) - a claim answers each output by publishing it at its exact path under its own epoch, so a restarted output never reaches a returning reader as the old one's cache - [Malformed group FETCH](/quest/m1/group-fetch-malformed.md) - an object missing its Timestamp in a relay's group FETCH ends the whole TIMESCALE track, as subscribe and joining FETCH already do - [FFI shape](/quest/m1/ffi-shape/README.md) - the bindings mirror Rust's layers: net at the root, then media, json, flate, audio, and video namespaces built from the handle below @@ -84,6 +85,7 @@ above them. - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - a lite-07 SUBSCRIBE asks for past and live ranges with absolute or relative ends, in either order, and replaces FETCH; merged ranges never starve a subscriber; relays fill misses by range, including over moq-transport - [Live largest location](/quest/m1/largest.md) - `largest()` replaces `latest()` in Rust and JS as a live stream of the track's largest location, known before the first group from the publisher's report - [Live TS clock validation](/quest/m1/ts-live-clock.md) - resolve the local PCR-rate failure with reproducible timing evidence and CI controls +- [TS export start-up drift](/quest/m1/ts-drift-startup.md) - a TS export never exits on start-up latency from a source within tolerance, while a far-off source still fails within seconds (#5564) - [Binding panic supervision](/quest/m1/ffi-panic-supervision.md) - a panic in any moq-c or moq-ffi background task reaches the host as a panic, never a lost callback or Cancel - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause - [Plan: a public model](/quest/m1/model-public-surface.md) - every crate-private model item is made public or unnecessary, so a custom publisher or subscriber builds from the public model alone diff --git a/quest/m1/lite-setup-backport.md b/quest/m1/lite-setup-backport.md new file mode 100644 index 0000000000..92d9b94d88 --- /dev/null +++ b/quest/m1/lite-setup-backport.md @@ -0,0 +1,46 @@ +# [S] Backport the Lite SETUP violation closes to release + +## Goal + +On `release`, a moq-lite peer that repeats a SETUP Parameter ID or opens a +second Setup Stream gets the session closed with PROTOCOL_VIOLATION (0x3), as +`drafts/draft-lcurley-moq-lite.md` requires, so a later 0.14.x CLI and 0.3.x +moq-net patch pass moq-dev/smoke's moq-lite conformance rows +L06-7-3-MUST-112 and L06-6-3-1-MUST-092. + +## Plan + +Reported in [#5521](https://github.com/moq-dev/moq/issues/5521) against `moq` +0.14.4. Verified 2026-10-15 that `main` already closes both cases: + +- A second Setup Stream: `PeerSetup::claim` (`rs/moq-net/src/lite/setup.rs`) + claims the slot as soon as the stream type is Setup, from 7ddfa101b in + #4704. Tests: `duplicate_setup_closes_the_session`, + `accept_setup_refuses_two_incomplete_setup_streams`, + `accepted_lite_setup_refuses_a_second_setup_stream`. +- A SETUP that fails to decode (a duplicate parameter returns + `DecodeError::Duplicate`): the `UniServe` Err branch in + `rs/moq-net/src/lite/subscriber.rs` returns ProtocolViolation, from + 0f6a8d1f9 in #4704. No test sends a repeated Parameter ID. + +On `release`, `UniServe` aborts a stream that fails to decode and returns +`Ok`, and `peer_setup.set` overwrites a second SETUP. The server's +`accept_setup` path already maps a decode error to ProtocolViolation there. + +Decided 2026-10-15 (maintainer): + +- One PR onto `release` (CONTRIBUTING) that ports only the two closes: the + `PeerSetup` claim and the close on a SETUP decode error, with their tests. + Not the rest of #4704 (early stream retention) or #5550. +- Add a repeated-Parameter-ID regression test on both `main` and `release`, + one that fails without the close. Check whether js/net closes on a repeated + parameter too; if it does not, report it as a follow-up rather than + widening this backport. +- The crate patch bumps happen at the release cut, not in this PR. + +Public API: none. Wire: none (`release` starts enforcing rules the draft +already states). + +## Closes + +- [#5521](https://github.com/moq-dev/moq/issues/5521) - moq-lite-06: a repeated SETUP parameter or a second Setup Stream does not close the session diff --git a/quest/m1/test-flakes-2/ts-video-content.md b/quest/m1/test-flakes-2/ts-video-content.md index 0f827e1137..5610b4b8b4 100644 --- a/quest/m1/test-flakes-2/ts-video-content.md +++ b/quest/m1/test-flakes-2/ts-video-content.md @@ -55,6 +55,26 @@ failed 3 of 4 runs, in two other ways: (43 to 64 ms) vanished mid-stream with no warning anywhere, and the exporter's release clock reported `dropped=0`. +Then [#5565](https://github.com/moq-dev/moq/issues/5565), on 2026-10-15: +4 of 51 `ts.yml` runs between 2026-10-14 19:57Z and 2026-10-15 03:55Z failed +`video-content` the same way, two on branches that change nothing in moq-net, +moq-relay or moq-mux's consumer: 38634848576 (`fix/tokio-windows-refused-dial`), +38657903232 (#5553), 38662560784 (`quest/m1/papercuts-rs`) and 38677047938 +(#5548's branch). Their artifacts expire about 2026-10-22. Each `sub.log` has +`current group evicted; skipping to next buffered group track="0.avc3" +group=1 error=Hang(Moq(Stream(Old)))`, 17 to 21 ms after a video subscribe +at about 1.98 s, and the capture keeps pictures 25 to about 33, then 50 +onward. So the prefix is cut by a serving hop resetting group 1 with `Old` +when group 2 lands, which fits the `max_delay = 0` candidate below: at a +zero budget, `poll_expired_while_pending` (`rs/moq-net/src/lite/publisher.rs`) +convicts a group the moment its successor arrives. Still open: the subscriber +had read frames, so it should have raised its budget before group 2. The +`relay.log` and `pub.log` will show which hop reset the stream and whether the +update had reached it. #5548 moved the zero-budget conviction from the +successor's first frame to its arrival, essentially the same instant, so it +should not change the rate. Not reproduced locally: 0 of 15 on main +(3abf0161c, macOS arm64). + So in each case the checks saw real loss. The flake is what makes the exporter lose content, and the loss goes unreported. @@ -115,6 +135,10 @@ pre-IDR content in group 0 (the importer opens a group only at a keyframe). Public API: none expected; the drop count may surface in the exporter's summary. Wire: none. +## Closes + +- [#5565](https://github.com/moq-dev/moq/issues/5565) - flaky: `just test ts` loses most of the first video group when it is evicted as `Old` just after subscribing + ## Related - [#5484](https://github.com/moq-dev/moq/pull/5484) - removes the WebSocket-to-QUIC upgrade behind local mode 1 diff --git a/quest/m1/ts-drift-startup.md b/quest/m1/ts-drift-startup.md new file mode 100644 index 0000000000..e4376a7e54 --- /dev/null +++ b/quest/m1/ts-drift-startup.md @@ -0,0 +1,56 @@ +# [M] A TS export never exits on start-up latency + +## Goal + +`moq export ts` never exits with "the source's clock runs ... ppm off ours" +for a source within the 30 ppm tolerance, however its latency moves while +the export starts (queueing that builds, or a backlog that drains). A source +far off real time (the 220,000 ppm case) still fails within seconds. `just +test ts --full` stops failing this way. + +## Plan + +Reported in [#5564](https://github.com/moq-dev/moq/issues/5564): 2 of 10 local +`--full` runs on `main` (3abf0161c) and 1 of 51 CI runs exited within 30 s. +The first `TS export release clock` lines read wild drift (-155315, +32045 +ppm), and passing runs log none. + +From reading the code on 5beead687 (not reproduced): `Buffer::steer` +(`rs/moq-mux/src/jitter.rs`) takes one floor every 2 s, fits an upper convex +hull, and reads both the rate and `gap` (extrapolated to `now`) from it, with +no sample or significance gate. `lost` is measured against `startup`, the gap +at the second floor. The export bails when `|slow| > 30 ppm` and `lost` +passes half the delay (450 ms at `--full`'s 900 ms). So a latency ramp after +the second floor, on a 0 ppm source, can end the export. Before #5490 +(41f3087e3) the exit also required the steered drift to reach its limit, +which made start-up noise unable to end it; `--full` (#5516) landed half an +hour after #5490. + +Decided 2026-10-15 (maintainer): + +- Reproduce first with the issue's loop, and record which floors skew the + envelope, before choosing a mechanism. +- Fix the measurement; do not add a minimum span or sample count. That keeps + the decision recorded when #5490 removed `quest/m1/slow-source.md`. +- Never widen the 30 ppm tolerance, `--delay`, or the `lost` threshold. +- Candidate mechanisms, for the implementer to pick from the evidence: + judge `lost` from observed floors instead of the envelope extrapolated to + `now`; exit only when the rate's uncertainty excludes 30 ppm; keep + `startup` floating until the rate is significant; and treat a fast + source's positive `lost` (a growing buffer) differently from loss. +- Mocked-time regressions in `jitter.rs`, each failing without the fix: a + 0 ppm source whose queueing builds 300 to 500 ms over 5 to 20 s and then + holds, the same draining, and both again with three tracks shaped like + `--full` (video sent up to 0.65 s ahead). + `a_source_far_off_real_time_fails_within_seconds` keeps passing. + +Public API: none. Wire: none. + +## Closes + +- [#5564](https://github.com/moq-dev/moq/issues/5564) - flaky: `just test ts --full` exits on the release clock's first rate estimates + +## Related + +- [Live TS clock validation](/quest/m1/ts-live-clock.md) - the grader's PCR-rate verdict on `--live`, a different layer +- [TS decoded content](/quest/m1/test-flakes-2/ts-video-content.md) - the other TS CI flake, on content loss From f3bd816681cc387084932d6d90bd0752195a690c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 15 Oct 2026 08:58:16 -0700 Subject: [PATCH 2/2] quest: say the drift envelope needs two floors Co-Authored-By: Claude Opus 5.5 --- quest/m1/ts-drift-startup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/quest/m1/ts-drift-startup.md b/quest/m1/ts-drift-startup.md index e4376a7e54..960c1e25cb 100644 --- a/quest/m1/ts-drift-startup.md +++ b/quest/m1/ts-drift-startup.md @@ -17,8 +17,8 @@ ppm), and passing runs log none. From reading the code on 5beead687 (not reproduced): `Buffer::steer` (`rs/moq-mux/src/jitter.rs`) takes one floor every 2 s, fits an upper convex -hull, and reads both the rate and `gap` (extrapolated to `now`) from it, with -no sample or significance gate. `lost` is measured against `startup`, the gap +hull, and reads both the rate and `gap` (extrapolated to `now`) from it. A +rate needs only two floors, with no minimum span or significance gate. `lost` is measured against `startup`, the gap at the second floor. The export bails when `|slow| > 30 ppm` and `lost` passes half the delay (450 ms at `--full`'s 900 ms). So a latency ramp after the second floor, on a 0 ppm source, can end the export. Before #5490