diff --git a/quest/m1/README.md b/quest/m1/README.md index 18f4bd1d61..15fceed49e 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -105,6 +105,7 @@ lands before the cut. - [Decode errors end the reader](/quest/m1/decode-errors.md) - a container decode error fails the reader in Rust and JS instead of reading as loss - [Transcode finishes its group](/quest/m1/transcode-open-group.md) - `moq-transcode` never aborts a group a fetch's live loop opened - [Rust request credit](/quest/m1/rs-request-credit.md) - a moq-net request fails at once without stream credit instead of waiting +- [A stalled subscriber holds bounded state](/quest/m1/egress-held-state.md) - group machines and bytes a credit-withholding downstream makes a publisher hold plateau at what its credit and the latency budget allow, on lite and moq-transport - [qmux no-wait opens](/quest/m1/qmux-no-wait.md) - @moq/qmux rejects an over-limit create when waitUntilAvailable is false, like Chrome - [Random-loss audit](/quest/m1/random-loss-audit.md) - every decision random-loss gates drove is kept, reworked, or reverted on shaped paths - [Timed-out requests clean up](/quest/m1/legacy-request-timeout-cleanup.md) - on drafts 14-16, Rust and JS send PUBLISH_NAMESPACE_DONE or UNSUBSCRIBE_NAMESPACE for a timed-out request before its local reset diff --git a/quest/m1/egress-held-state.md b/quest/m1/egress-held-state.md new file mode 100644 index 0000000000..d77f3b6092 --- /dev/null +++ b/quest/m1/egress-held-state.md @@ -0,0 +1,120 @@ +# [L] A stalled subscriber holds bounded publisher state + +## Goal + +What a downstream peer can make a publisher hold is bounded by the protocol's +own shape, not by how long it stalls. A subscriber that withholds stream +credit, or drains slower than the media, across as many tracks as its session +limits admit, makes a relay (or any moq-net publisher) hold group state and +bytes that plateau at what the peer's credit and the latency budget allow, +on both moq-lite and moq-transport. A healthy viewer behind a congested link +still gets the freshest groups. + +Non-goals: capping sessions or connections (see Plan), ingress memory +([Bound incomplete messages](/quest/m1/quic/receive-memory.md) owns it), and +new limit constants that only refuse input. + +## Plan + +Planned 2026-10-19 after the egress audit in #5958, under the rule that +network input is untrusted and its cost is bounded by design (#5961). Facts +as of 83805955a; re-verify, they move. + +Where it stands: + +- Each group a subscription reaches becomes its own serve machine at once, + and waits in `Open` for stream credit with no in-flight bound: lite + `TrackRun::serve` (`rs/moq-net/src/lite/publisher.rs`, around 3722), IETF + `TrackServe` ("one machine per group, with unlimited concurrency", + `rs/moq-net/src/ietf/publisher.rs`, around 4002). +- A parked machine leaves only when its group expires. An aborted group never + does: `expiry_pending` is false once aborted (`model/group.rs`, around 1924 + and 2174), and the `Open` arms check only expiry. The pool's idle window + (30 s by default) evicts the unread group's bytes, so the machine outlives + its content and parks forever. When credit finally arrives, it may be spent + on a group that can no longer be served. +- The serving budget is effectively unbounded. A peer whose wire carries no + max delay (lite-01/02, every moq-transport draft) is served at about 2^53 ms + (`MAX_SAFE_DELAY_MS` in both publishers), and a lite-03+ peer names any + value it likes. Locally only the origin's `cache_duration` (`Duration::MAX` + unless the relay sets `--cache-duration`) or the publisher's `max_age` + clamps it. +- Session limits cap subscriptions and fetches (`rs/moq-relay/src/limits.rs`: + 1000 subscriptions and 10000 fetches per client session), never groups in + flight per subscription. + +Abuse scenarios to cover: + +1. A subscriber at its subscription cap never grants uni-stream credit. Held + machines grow with every group published, forever, even after the pool + frees their bytes. +2. The same peer with an inert pool (no capacity, no idle window, as a library + user may configure). The pool keeps every group by policy, so judge only + what the subscriber adds: held state beyond a no-subscriber baseline. +3. A peer grants stream credit but withholds byte credit, so opened streams + pin their groups past the point they are useful. +4. A lite-03+ peer asks for an enormous max delay so nothing ever expires. +5. The benign twin: a real viewer on a bufferbloated link. The fix must not + cost it the live edge. + +Recommended directions (the implementer settles the shape): + +- Make a group's serve state follow credit rather than arrival. One option is + to open a stream only when the session can grant one and then pick the best + group still worth sending at that moment, so a parked backlog lives in the + cache (already billed and evicted) rather than in per-subscriber machines. + Whatever the shape, cross-subscription priority and per-broadcast fairness + ([Scope track priority](/quest/m1/track-priority-scope.md)) must still + decide who gets the next stream. +- A parked group that can no longer be served (aborted, evicted, expired) + leaves at once and is named to the subscriber where the version can, through + [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md). +- The serving budget never exceeds what this hop would retain anyway: clamp + the requested or defaulted max delay by the local retention (the + `cache_duration` ceiling, the track's `max_age`, and possibly the pool's idle + window). Consider whether the relay's default `cache_duration` should follow + the pool's window instead of `Duration::MAX`; that changes a user-facing + default, so update `doc/bin/relay/` with it. +- Prefer bounds the protocol already carries (stream credit, max delay, + `max_age`, the cache pool) over a new per-subscription constant. If a number + is unavoidable, it must come from one of those. + +Sessions: capping them is out of scope. A session costs a handshake, auth, and +the per-session limits above, and a count cap would be a constant that only +throws. Admission control (per-token or per-address session caps) belongs to +the deployment's auth server, which already refuses a connect with a 401 or +403 ([relay auth](/doc/bin/relay/auth.md)); this repo needs nothing more. + +Reproduce first. A mock-time sim test (or a `just` drill against a real relay) +with a subscriber that withholds stream credit, or drains below the media rate +behind a deep finite queue, while the publisher writes groups across many +tracks. Measure, over time, live serve machines (pending and opened, as a +gauge) and the bytes they retain outside the pool (batches, chunks, and +unsent transport data), beside pool-billed bytes compared with a +no-subscriber baseline. Only lite counts pending opens today; IETF +`TrackServe` counts streams only after they open, so add the gauge there. On +main they grow with time; done, they plateau. Per the fan-out benchmark rule, +sweep tracks by groups per track, and subscribers by tracks, timing each +stream grant too, so a cost that grows with the table (such as scanning every +subscription when one stream frees up) shows up as a slope. Cover lite and +moq-transport, and scenario 5 still receives the latest group. +Include a benign withholder: a peer bounded by +[Bound incomplete messages](/quest/m1/quic/receive-memory.md) withholds credit +by design under memory pressure, so two relays running both quests must still +plateau and keep the live edge. + +Done looks like: those measurements plateau on both protocols and every +scenario above, the regression tests fail on main and run in CI, the JS +publisher in `js/net` is checked for the same shape and fixed or its gap +named in the PR, and the PR reports public API and wire impact (expected: +none on the wire; possibly a relay default). + +## Related + +- [Bound incomplete messages](/quest/m1/quic/receive-memory.md) - the ingress counterpart: a receiver withholds QUIC credit to bound what it holds, and this quest bounds what the sender holds while it does +- [Rust request credit](/quest/m1/rs-request-credit.md) - requests fail fast without stream credit; publisher group streams are left to this quest +- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - how a publisher names a group it gave up on +- [Scope track priority](/quest/m1/track-priority-scope.md) - who gets the next stream across subscriptions and tenants +- [Public serving](/quest/m1/public-model/serve.md) - makes group expiry while serving public; coordinate the publisher serve loops with it +- [Treat network input as untrusted](https://github.com/moq-dev/moq/pull/5961) - the rule this quest applies +- [Resumed group head](https://github.com/moq-dev/moq/pull/5958) - the audit that found this, and bounded held heads by the cache pool