From 4167479cd51d6acdd0e3643c2332a4ffe75d98e4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 10 Oct 2026 20:22:28 -0700 Subject: [PATCH] docs(quest): plan flow-controlled receive memory Co-Authored-By: GPT-6 --- quest/m1/ietf-dispatch-headers.md | 15 ++++- quest/m1/quic/README.md | 13 +++- quest/m1/quic/peer-limits.md | 10 +++- quest/m1/quic/receive-credit.md | 58 ++++++++++++++++++ quest/m1/quic/receive-memory.md | 99 +++++++++++++++++++++++++++++++ quest/m2/rate-quic.md | 46 ++++++++++---- 6 files changed, 223 insertions(+), 18 deletions(-) create mode 100644 quest/m1/quic/receive-credit.md create mode 100644 quest/m1/quic/receive-memory.md diff --git a/quest/m1/ietf-dispatch-headers.md b/quest/m1/ietf-dispatch-headers.md index 9a019dbf74..4e4468e854 100644 --- a/quest/m1/ietf-dispatch-headers.md +++ b/quest/m1/ietf-dispatch-headers.md @@ -21,7 +21,18 @@ this is Rust only (decided 2026-10-08). Test, beside #5086's `a_silent_stream_does_not_hold_up_the_next`: a silent bidi request stream does not block a SUBSCRIBE on the next one. -`release` has the same loop; once this lands on `main`, backport it the way -#5125 carried the uni half. +Review of [#5208](https://github.com/moq-dev/moq/pull/5208) found that +concurrent partial headers return transport credit before parsing completes, +allowing retention far beyond the receive window. Keep that PR held until +the shared native receive-memory work below bounds headers and frame data; +do not substitute a separate header quota or speculative-allocation fallback. + +`release` has the same loop. Its backport needs a separate compatibility +assessment because the new receive-credit prerequisite targets the in-tree +stack; this plan does not authorize moving that stack onto `release`. Public API: none. Wire: none. + +## Required + +- [Bound message assembly](/quest/m1/quic/receive-memory.md) - concurrent headers retain connection credit and pressure cancellation restores progress diff --git a/quest/m1/quic/README.md b/quest/m1/quic/README.md index 62ec5a0624..26b021ef2a 100644 --- a/quest/m1/quic/README.md +++ b/quest/m1/quic/README.md @@ -7,7 +7,8 @@ forked from quinn (see [the fork](/quest/m1/quic/fork/README.md) for why quinn and not noq). One core serves the tokio backend, the thread-per-core `moq-uring` backend, and qmux; iroh stays on upstream noq. The features are BBR correctness, reliable stream resets, hierarchical stream scheduling with -per-broadcast fairness, wider limits for relay peers, and endpoint sharding. +per-broadcast fairness, bounded incomplete-message memory, wider limits for +relay peers, and endpoint sharding. Per-stream acknowledgment progress, per-stream deadlines, qmux on the shared stream state machine, and the experiments (the egress profile, ECN measurement, deadline keep-alive) live in [m2](/quest/m2/README.md); media @@ -56,9 +57,19 @@ consumes them. Decided 2026-10-08: L4S and careful resume moved on to m3, and GCC shrank to a receive-timestamps spike, since none has a consumer. The BBR app-limited fixes land in `moq-quic` without waiting for the switch. +Decided in the 2026-10-10 receive-memory plan: native message assembly holds +connection credit until complete-unit handoff, while stream credit advances +on reads with a very large configurable default. MoQ recovers under assembly +pressure with receiver STOP_SENDING, including against JavaScript senders. +The memory allowance must fit the largest supported encoded unit. Control +streams have highest priority but are not immune to cancellation. Completed +cache limits remain separate; bitrate pacing composes with this memory cap. + ## Required - [Hard fork](/quest/m1/quic/fork/README.md) - quinn in-tree as `moq-quic`, with BBR3 and lazy stream slots, and MoQ switched onto it +- [Retain connection credit](/quest/m1/quic/receive-credit.md) - native receivers own connection credit independently of stream reads +- [Bound message assembly](/quest/m1/quic/receive-memory.md) - complete headers and frames return credit, and receiver priority cancellation restores progress - [BBR idle burst](/quest/m1/quic/bbr-app-limited.md) - a fork regression proves a burst after a long idle is paced at the learned bandwidth, closing #4219 - [Mark BBR starvation wherever the source runs dry](/quest/m1/quic/bbr-app-limited-edges.md) - partial polls count, local send caps do not, receiver credit is pinned - [Reliable stream reset](/quest/m1/quic/reliable-reset.md) - `RESET_STREAM_AT`, diff --git a/quest/m1/quic/peer-limits.md b/quest/m1/quic/peer-limits.md index dcfec85634..f26f7dfff5 100644 --- a/quest/m1/quic/peer-limits.md +++ b/quest/m1/quic/peer-limits.md @@ -13,8 +13,8 @@ stay small so one client cannot reserve a relay's memory. The in-tree `moq-quic` `Connection` (`rs/moq-quic/src/connection/mod.rs`) already exposes runtime `set_max_concurrent_streams`, `set_receive_window`, and `set_send_window`; a raise queues `MAX_STREAMS` and `MAX_DATA` on the next -packet, and a shrink is a debt paid as the peer consumes credit. The core -needs no change; the work waits only for +packet, and a shrink is a debt paid as the peer consumes credit. The existing +setters provide this seam; the transport integration waits for [the fork switch](/quest/m1/quic/fork/README.md), so moq-tokio and moq-uring run on `moq-quic` instead of `moq-noq`. @@ -35,6 +35,12 @@ surface with the QUIC values rather than adding its own. - Refuse a `peer` value below the client default rather than silently shrinking. +The [receive-memory plan](/quest/m1/quic/receive-memory.md) requires client +and peer memory allowances to fit the largest supported encoded unit, +including framing. Preserve that validation and retained-credit accounting +when applying runtime limits; wider stream counts must not create a second +unbounded assembly pool. + Tests: a cluster session sees the raised `MAX_STREAMS` after SETUP and a viewer session does not; the io_uring path applies the same values; a `peer` table below the defaults is refused at resolve time. diff --git a/quest/m1/quic/receive-credit.md b/quest/m1/quic/receive-credit.md new file mode 100644 index 0000000000..c2bd474fbb --- /dev/null +++ b/quest/m1/quic/receive-credit.md @@ -0,0 +1,58 @@ +# [L] Return connection credit when the receiver releases bytes + +## Goal + +Native receivers can read QUIC stream bytes without returning connection +credit until their consumer releases those bytes. The same ownership works +through the tokio and io_uring adapters, including cancellation and teardown. +Stream credit remains independent, so a finite stream window does not force +an entire application message to fit before more bytes arrive. + +## Plan + +Decided in the 2026-10-10 receive-memory interview: use `moq-quic` after the +fork switch, not a new feature in the frozen moq-dev/noq fork. This quest +provides the transport primitive; [message assembly](/quest/m1/quic/receive-memory.md) +activates it with the recovery policy needed for progress. + +Today `Chunks::finalize` returns both kinds of credit before the parser has +a complete message. Separate returning borrowed transport state from +returning connection credit. Carry ownership through the native adapters +without copying payloads solely to track credit. Prefer an owned handle or +explicit consumption accounting over callbacks; keep the surface limited to +what the message-assembly consumer needs. + +Replenish `MAX_STREAM_DATA` on reads. Defer `MAX_DATA` until the owner releases +the bytes, exactly once, even after FIN, STOP_SENDING, RESET_STREAM, or stream +state reclamation. Unread discarded bytes, late data on stopped streams, +reset final-size gaps, and already-read retained bytes must not leak or +return the same credit twice. Releasing useful credit must eventually wake +a blocked peer even below the normal window-update threshold; do not depend +on optional DATA_BLOCKED frames. Runtime window shrink debt still applies. + +Expose enough credit accounting and events for MoQ to detect assembly +pressure. QUIC performs cancellation but does not choose application victims. +A sender's write being blocked, or its data being acknowledged, does not +prove receiver assembly deadlock. No sender-side shedding is added. + +Tests use the transport simulator and controlled time: stream credit advances +while connection credit stays held, explicit release resumes the peer, and +all terminal paths account once even when retained bytes outlive the stream. +Cover a useful release smaller than the update threshold and compose with +reliable reset when available. Wire adapter tests into CI and measure the +accounting cost across stream counts and payload sizes. + +Public API: a native receive-credit ownership/observation seam, finalized +with its consumer. Wire: existing QUIC flow-control and cancellation frames; +no new MoQ message or negotiation. Browser and iroh receivers do not gain a +capability their transports cannot provide. + +## Required + +- [Switch MoQ onto moq-quic](/quest/m1/quic/fork/switch.md) - both native adapters use the in-tree transport + +## Related + +- [Bound message assembly](/quest/m1/quic/receive-memory.md) - retains connection credit until a complete message is handed off +- [Reliable stream reset](/quest/m1/quic/reliable-reset.md) - compose retained credit with reliable prefixes and final-size accounting +- [Relay peers get wider limits](/quest/m1/quic/peer-limits.md) - runtime receive-window changes preserve retained-credit accounting diff --git a/quest/m1/quic/receive-memory.md b/quest/m1/quic/receive-memory.md new file mode 100644 index 0000000000..ad6267897e --- /dev/null +++ b/quest/m1/quic/receive-memory.md @@ -0,0 +1,99 @@ +# [XL] Bound incomplete messages with QUIC connection credit + +## Goal + +Native MoQ receivers bound incomplete frames and protocol messages by the +connection memory allowance, returning connection credit only when each +complete unit is parsed and handed off or discarded. When concurrent partial +messages prevent progress, the receiver cancels streams by MoQ priority and +continues. This works with Rust and JavaScript senders through tokio and +io_uring; completed media caching remains separately governed. + +## Plan + +Decided in the 2026-10-10 interview: cover headers and frame/group data alike. +The frame preallocation budget only bounds speculative allocation, and the +cache has soft limits; neither bounds aggregate incomplete messages. Use +peer-visible QUIC flow control instead of a separate header quota error. +The native transport primitive is a separate prerequisite; do not activate +whole-message retention without its recovery policy in this quest. + +Hold connection credit through reader buffers, type/length prefixes, headers, +extensions, and frame assembly. Attribute read-ahead bytes to the next unit +rather than releasing them with the completed one. Release on complete-unit +handoff, not after application playback or cache eviction. Avoid retaining +an entire multi-frame group as one unit. Account for received-but-unread and +out-of-order transport bytes as well as parser-held bytes; they share the +same allowance. Speculative allocations and copies need separate accounting +or bounds: QUIC byte credit is not a claim about total process RSS. + +Keep the existing stream-window setting and honor explicit overrides. +Give the native path a very large default so it normally adds no round trips; +stream credit is replenished on reads, independently of message completion. +Do not remove or ignore the setting without benchmark evidence and a new +maintainer decision. The connection memory allowance is the shared cap. + +Require that allowance to fit every supported encoded message, including +framing and extensions. Validate configuration and align native defaults +rather than discovering an impossible frame after receiving it. The current +32 MiB frame limit exceeds io_uring's 16 MiB connection default; a legacy +64 MiB announcement body also needs room for framing beyond 64 MiB. Derive +the minimum from the supported units, not just the payload limit. Lowering +a protocol/frame size limit is a separate maintainer decision, not an implicit +way around this requirement. Runtime memory-limit updates obey the same +minimum; already-advertised credit cannot be revoked. A rate limit can pace +a smaller amount of outstanding credit while the memory allowance remains +large enough for the complete unit. + +MoQ selects victims; the transport only reports pressure and sends +STOP_SENDING. Do not add sender-initiated shedding. This keeps recovery +available with browser senders and avoids treating intentional rate pacing +as receiver deadlock. The sender's required RESET_STREAM response is still +normal QUIC behavior, not a second recovery policy. + +Use the receiver's assembly and credit accounting to recover when partial +units exhaust progress capacity. Do not require peer DATA_BLOCKED, an +arbitrary timeout, loss, or a deadline. Demonstrate that useful work can +complete after cancellation, including when incoming streams keep arriving; +merely cycling through victims is not recovery. Distinguish bytes still able +to arrive under advertised credit from memory held by incomplete units. + +Within the existing broadcast fairness domains, cancel lower-priority media +and less useful groups first, respecting subscription priority and group +order. Prefer low-priority attributed media over unidentified headers so +headers have a chance to identify themselves; unidentified streams remain +eligible if they prevent progress. Control streams have the highest priority +but are eligible too. If canceling an essential control stream terminates the +session under its protocol, surface that outcome instead of leaving tasks +parked. There is no permanently protected class or separate control reserve. +Use deterministic tie-breaking without comparing unrelated broadcasts' raw +priority values. Do not invent another subscription-priority API here. + +Regression coverage must include many partial headers and frames filling the +shared allowance, a finite stream window smaller than a frame, the largest +supported encoded unit, read-ahead across unit boundaries, cancellation and +late reset/data races, unidentified and control-only pressure, and continued +progress after selecting victims. Exercise a JavaScript sender against both +native receive paths and run all-language interop. Measure retained bytes, +throughput, and completion latency across concurrent streams and frame sizes, +including finite versus very large stream windows. Wire checks and benchmarks +into CI; no wall-clock sleeps in unit tests. + +Public API: native receive-credit integration and validation of the existing +window configuration; no new header-specific quota. Wire: existing QUIC +flow-control and STOP_SENDING/RESET_STREAM behavior, no MoQ format change. +Browser/iroh receive-side guarantees and completed-cache limits are outside +this quest. Update existing configuration and transport docs inline; no +separate guide quest is needed. + +## Required + +- [Retain connection credit](/quest/m1/quic/receive-credit.md) - core and native adapters support separate byte ownership and stream-credit release + +## Related + +- [IETF request headers](/quest/m1/ietf-dispatch-headers.md) - concurrent request parsing must not amplify unbounded partial-header retention +- [Scope track priority](/quest/m1/track-priority-scope.md) - broadcast fairness domains and subscription priority semantics +- [Hierarchical stream scheduling](/quest/m1/quic/scheduler.md) - share priority semantics without requiring sender-side recovery +- [QUIC bitrate caps](/quest/m2/rate-quic.md) - pace credit grants inside the independently bounded memory allowance +- [Relay peers get wider limits](/quest/m1/quic/peer-limits.md) - client and peer windows must both fit supported units diff --git a/quest/m2/rate-quic.md b/quest/m2/rate-quic.md index 15b3d653ca..269d7fca50 100644 --- a/quest/m2/rate-quic.md +++ b/quest/m2/rate-quic.md @@ -13,21 +13,33 @@ Decided in the 2026-10-04 plan: enforce in the QUIC stack, through backpressure the peer cannot ignore, rather than metering and closing. - **Ingress: pace flow-control credit.** The relay extends `MAX_DATA` at - `rate`, with a window of about one second of `rate` as the burst. A peer + `rate`, with about one second of `rate` as the burst allowance. A peer that sends past its credit violates flow control and the connection fails, so a modified client cannot exceed the cap. An honest one goes flow-control limited (its congestion controller does not back off) and queues until [the grant](/quest/m2/rate-grant.md) clamps its encoder. - Per-stream windows are unchanged; the connection credit is the cap. + Keep rate pacing separate from the connection memory allowance in + [message assembly](/quest/m1/quic/receive-memory.md). The memory allowance + must fit the largest supported encoded unit even when it takes many rate + intervals to receive. Limit each grant by both rate allowance and available + memory backing; do not wait for a whole-message credit release to grant + previously unadvertised headroom. Track retained received bytes separately + from outstanding granted credit, bounding their combined memory commitment. + Bound unspent grants as well, so an idle peer cannot bank a memory-sized + burst. A large frame must finish gradually without enlarging the rate burst. + Stream-credit replenishment remains independent. - **Datagrams draw from the same allowance.** DATAGRAM frames are not flow controlled, so credit alone does not bound them. Received datagram payload is charged to the same allowance that paces `MAX_DATA` credit, and a datagram past the cap plus burst is dropped before routing, the same as - network loss. Charge on receipt (stream bytes consumed plus datagram - payload), so datagram bytes withhold the next credit extension: unspent - stream credit does not starve datagrams, a publisher mixing both under the - cap loses none, and mixed traffic overshoots by at most one window. An - honest publisher is never disconnected, so this does not wait for + network loss. Charge newly received stream bytes and datagram payload + once, not application message completion or playback, so datagram bytes + withhold the next credit extension: unspent stream credit does not starve + datagrams, a publisher mixing both under the cap loses none, and mixed + traffic overshoots by at most the outstanding burst credit. Cancellation + frees memory but does not refund bytes against the rate allowance. An + honest publisher is never disconnected for exceeding its rate, so this + does not wait for [the grant](/quest/m2/rate-grant.md). - **Credit cannot be retracted** (RFC 9000 ยง4.1). Every session, capped or not, starts with an `initial_max_data` shrunk to what the handshake, @@ -35,11 +47,11 @@ backpressure the peer cannot ignore, rather than metering and closing. cannot bank a full default window and spend it after a low cap arrives. After auth the relay raises it through the `set_limits` seam: the normal window for an uncapped session, paced credit for a capped one. A lowered - cap (revalidation, a union shrinking) uses `moq-quic`'s shrink-as-debt - `Connection::set_receive_window` (`rs/moq-quic/src/connection/mod.rs`), described in - [peer limits](/quest/m1/quic/peer-limits.md), so the overshoot is - bounded by the credit outstanding at the change and the test asserts that - bound. + rate (revalidation, a union shrinking) withholds future grants against + outstanding credit rather than shrinking assembly memory below a legal + message. Memory-window reductions retain the shrink-as-debt semantics + described in [peer limits](/quest/m1/quic/peer-limits.md). Overshoot is + bounded by credit outstanding at the change and the test asserts that bound. - **Egress: cap the pacer.** The send rate is `min(controller rate, cap)`, so a subscriber below the broadcast's bitrate gets MoQ's normal group skipping, not a growing queue. @@ -61,10 +73,18 @@ unspent pre-auth credit, and one whose cap drops with credit outstanding, stay within the stated bound; an uncapped session gets its normal window after auth; egress to a capped subscriber never exceeds the cap; raising and lowering the cap on a live connection takes effect; the io_uring path does -the same. +the same. Also prove that a frame larger than one rate interval completes +within the memory cap, an idle sender cannot accumulate a larger burst, +assembly cancellation does not refund rate charges, and intentional rate +pacing alone does not trigger pressure cancellation. + +Decided in the 2026-10-10 receive-memory interview: memory allowance and +rate pacing are independent bounds. This replaces the earlier assumption +that a one-second rate window can also be the complete assembly allowance. ## Required - [Hard fork](/quest/m1/quic/fork/README.md) - the credit and pacer changes land in `moq-quic` - [Peer limits](/quest/m1/quic/peer-limits.md) - adds the runtime `set_limits(Limits)` seam and shrink-as-debt behavior this extends +- [Bound message assembly](/quest/m1/quic/receive-memory.md) - separate retained memory from paced outstanding credit - [Bitrate claim](/quest/m2/rate-claim.md) - the cap this enforces