feat(quic)!: bound incomplete messages with receive leases by kixelated · Pull Request #5878 · moq-dev/moq

kixelated · 2026-10-17T17:31:26Z

Supersedes #5735, which stays open as a reference draft. This bounds incomplete messages with a receive lease and completes quest/m1/quic/receive-memory.md. It also carries the Bun sender, the receive-credit tests, and benchmark_receive_windows from #5752. #5903's driver decoupling builds on it.

What changes

Bytes now stay in the transport, and moq-net never holds credit. A reader leases a stream's unread bytes as a bytes::Buf, advances past what it parses, and returns the credit with release() once a whole unit is handed off. The connection window therefore bounds every incomplete header and frame.

The connection window can fill with incomplete messages that none can finish. When that happens, the tokio and io_uring drivers evict the lowest-ranked stream, the newest one on a tie. Eviction drops the stream's buffered bytes, returns their credit through MAX_DATA, and fails the stream's next lease with Evicted. moq-net then stops the stream with its own code, which returns the credit of any bytes it had already taken; until that stop, nothing else is evicted. There is no timeout and no new peer signal.

Nothing is evicted while the peer still has connection credit, that is, while the leased and unread bytes of every stream stay below the connection window. A receiver that sizes its window above what its incomplete units can hold never sees an eviction, and CreditStats::evicted confirms it.

A frame larger than the window waits like any other partial unit, so the stuck window evicts it. Only a single want past the window is refused with TooLarge.

Public API

moq-net transport::poll:

pub trait RecvStream: 'static {
    type Error: Error;
    type Lease<'a>: Lease where Self: 'a;
    fn poll_lease(&mut self, cx: &mut Context<'_>, want: usize) -> Poll<Result<Self::Lease<'_>, Self::Error>>;
    fn set_priority(&mut self, priority: i64); // higher keeps the stream longer
    fn stop(&mut self, code: u32);
    fn poll_read_chunk(&mut self, cx, max) -> ...; // provided
    fn read_chunk(&mut self, max) -> ReadChunk<'_, Self>; // provided
}
pub trait Lease: Buf {
    fn chunks(&self) -> impl Iterator<Item = &[u8]>;
    fn is_fin(&self) -> bool;
    fn release(self); // returns the credit of every byte advanced or copied out
}
pub trait Chunks: 'static { poll_read_chunk, stop }  // plain readers
pub struct Buffered<S: Chunks>; pub struct BufferedLease<'a>;
pub const MIN_RECEIVE_WINDOW: u64;

Removed from RecvStream: poll_read, poll_read_buf, poll_closed, read, read_buf, closed, Read and ReadBuf. A lease with is_fin replaces the end observation.
advance and copy_to_bytes keep credit until release. Dropping a lease releases nothing. A want beyond the stream's window is an error, not a wait.
iroh, WebAssembly, qmux (WebSocket, TCP, WebRTC), FFI and the memory transport implement Chunks and read through Buffered, which bounds nothing (documented).
New CloseKind::Evicted, which maps to the existing Error::Evicted.
The receive ranks (RECV_CONTROL, recv_group) are crate-private.
moq-tokio: the driver RecvStream::poll_lease and set_priority, and driver::Lease<'a>. The quic web RecvStream implements the moq-net trait natively. Deleted read_held, poll_read_held and driver::Credit.
moq-uring: quic::RecvStream::poll_lease and set_priority, and quic::Lease<'a>. Deleted Credit, poll_read_held and the read-ahead backlog. Deleted Connection::stats(); the stats go through the moq-net trait. Added Error::{Evicted, TooLarge}.
moq-quic: RecvStream::{lease, take, advance, release, set_priority}, Leased, LeaseError and Streams::leased. Deleted the public Credit, Chunks::hold (now a private withhold) and Connection::release. CreditStats is now { leased, available, evicted, peak_spent }.
moq-net: transport::Stats::streams_evicted() (provided, None) and session::Stats::streams_evicted, filled by the tokio and io_uring QUIC backends.
The tokio and io_uring driver Lease types implement Buf.
The Grouped transport trait (tokio and uring) gains type Recv; fn recv(...).
MIGRATE.md entries for moq-net, moq-tokio and moq-relay. The moq-quic and driver changes replace APIs that never shipped in a release.

Configuration:

The tokio DEFAULT_RECEIVE_WINDOW and the uring driver's connection window are 128 MiB; they were 64 MiB and 16 MiB. The stream window (4 MiB) and send window (16 MiB) defaults are unchanged.
A connection window below MIN_RECEIVE_WINDOW is refused at config load, including [peer.quic].
The peer default stays at 640 MiB.

Wire impact

None to message framing or codes. A receiver may now STOP_SENDING an evicted stream with moq-net's existing Evicted code. No draft changes.

Receive ranks

Control streams rank i64::MAX.
A data stream ranks 0 until its header is decoded.
After that it ranks recv_group(priority, sequence, order), below every unidentified stream. The lowest subscription priority goes first, then the group its subscription would deliver last: the oldest under the default newest-first order, the newest under an ascending one.

Iteration 2026-10-20

Merged main three times. The fallout: SendOptions on open_uni/open_bi, Checkpoint replaced by offsets, the fuzz target's tampered-packet checks (now applied to leased streams too), and an ietf adapter test reading a raw virtual stream.
#5908 queues MAX_STREAMS when a stop frees a stream. An eviction that frees one now does the same, with a regression test that fails without it.
#5979 removed webtransport.protocols, which broke both Bun arms. The sender now passes versions.
The receive rank now follows the subscription's group order, as the sender's rank does.
zshenker's review: CreditStats::evicted, the no-eviction contract on lease() and the drivers' poll_lease, and an Evicted note to stop with your own code, since dropping a driver stream sends 0.
Bufferbloat: a shaper with a bottleneck below the sender's rate and a 250 ms finite queue, with no random loss. lite_partial_frame_gives_way_behind_bufferbloat recovers through it, bufferbloat_is_not_a_stuck_window shows pacing alone evicts nothing, and benchmark_receive_windows runs its grid on loopback and through a 200 Mbit/s bottleneck, failing on any eviction. It gets a five-minute nextest budget, like benchmark_ffi.
Deleted the finished quest. Main had retired the m1 QUIC epic's other child, so the epic is complete and goes too.
Codex: an eviction whose freed credit only paid down a shrunk window's debt advertised no MAX_DATA and stopped there, leaving the other reader and the peer waiting on each other. Eviction now continues until the peer has credit again; stuck_window_evicts_past_shrink_debt fails without it. LeaseError stays exhaustive like moq-quic's other stream errors (reply on the thread).
Codex: a stuck pass ranks its candidates once into a heap, so evicting many streams to pay debt stays linear. Eviction keeps waiting for a woken reader that has not leased yet, as it does for one that never leased; lease() now documents that a reader giving up on a stream must stop it.
Codex: eviction returned the credit of bytes a reader had taken while the reader still owned them, so a hostile peer could refill the window before the reader dropped them. Taken bytes now keep their credit until the stream is stopped, eviction pauses meanwhile, and the stop reruns the stuck check.

Iteration 2026-10-21

Merged main three times. #5903 landed first, and the m1 epic README conflict resolved by deleting it. #5909 resets a send stream dropped unfinished, so the driver eviction test now finishes its writers.
Lease is a bytes::Buf plus chunks, is_fin and release. Parsers use remaining, chunk, advance and copy_to_bytes as with any buffer, and release() replaces consume(n), so a reader can no longer pass a count that disagrees with what it parsed. LeaseBuf is gone. moq-quic mirrors it with advance and release.
Codex: a release that only paid shrink debt never reran the stuck check. Releasing, dropping and stopping now share one unstick(); consume_into_shrink_debt_evicts fails without it. The arrival-path MAX_STREAMS finding is declined: process_payload queues it after every packet.
Codex, second round: a stopped stream's arrival could return credit that only paid shrink debt without rechecking the window, a reset that woke a waiting lease was missing from the unlooked set, and a reset's final size never counted toward peak_spent. STREAM and RESET_STREAM now share one arrival tail; stopped_arrival_into_shrink_debt_evicts, reset_wake_is_not_stuck and reset_counts_toward_peak each fail without it.
A lease waiting on more bytes copies its slices out of their datagram buffers, so a few bytes no longer pin a whole GRO batch (waiting_lease_releases_datagram_buffers).
The stuck check is O(1) per wait: an unlooked set and an evicted-bytes counter skip the full scan, which stuck_check_stays_linear bounds at 512 streams. It was O(N) per wait, O(N²) over a burst.
CreditStats::peak_spent records the most credit the peer had spent at once. moq-net exposes evictions as Stats::streams_evicted, so a deployment can see a window that is too small. The relay's /metrics does not export it yet.
New criterion benches in moq-quic: receive_lease (lease vs read, 1-256 streams × 2-16 packets) and receive_evict (16-4096 streams stuck at once), both nightly.
benchmark_receive_windows varies one setting at a time from a default cell, adds a connection-window axis, keeps up to 1 MiB in flight per track instead of one group, and reads the peak from peak_spent instead of a 1 ms sampler.
New quest One receive budget per worker plans the QUICHE-style shared buffer, as decided below.
Merged main again. Main's stream credit on the memory transport now lives on its Chunks reader, and the receive-credit paths move into release-gate.yml, which now selects the interop workflow for release PRs. A driver doc link still named the deleted read_held.

Iteration 2026-10-22

Merged main (clean). The review of 82bd6fc never finished, so Codex reviewed 8b45473 instead.
CodeRabbit: stop on a lease that outlived its finished stream returned ClosedStream before rerunning the stuck check, so credit that only paid shrink debt could stall the window. It now reruns unstick() and succeeds; stop_finished_lease_into_shrink_debt_evicts fails without it.
CodeRabbit: moq-tokio's MIN_RECEIVE_WINDOW assertion is gated on _transport, so the nightly per-feature check's driver arm compiles again. The receive-credit step joins the interop report's inventory.
Codex: a reader waiting behind a gap now holds eviction off, since retransmitting the gap spends no credit. Before, ordinary loss on a full window could evict a group the retransmission would have completed. gap_is_not_a_stuck_window fails without it.
Codex, declined: Buffered gets no Drop. The wrapped Chunks stops on drop, as each did when it implemented RecvStream; Chunks::stop now says so.
Codex: the receive rank compares group sequences across subscriptions of equal priority. Kept as is (decision 13).

Iteration 2026-10-22 (merge)

Merged main twice. The only conflicts were in the quest tree: main edited a link in the m1 QUIC epic this PR deletes (kept deleted, decision 8), and the m1 milestone list (took main's entries, minus the epic). No code conflicts; main's QUIC changes (#6146, #6201) are send-side and handshake fixes, and #6214 is JS-only.
Decision 13 settled by the maintainer: option A.

Benchmarks

benchmark_receive_windows, release, medians of 3. finite keeps the driver's 4 MiB stream window; follow gives each stream the connection window. Peak is CreditStats::peak_spent. No run evicted a stream, and the bottleneck's queue never overflowed. Loopback throughput varies about 2x between runs on this shared machine; the bottleneck rows are stable.

path	axis	streams	frame	window	stream	peak MiB	MiB/s	p50 ms	p99 ms
loopback	default	16	256 KiB	128 MiB	finite	6.2	482	28	95
loopback	streams	1	256 KiB	128 MiB	finite	1.0	282	2.7	13
loopback	streams	256	256 KiB	128 MiB	finite	15.2	161	1033	1588
loopback	frame	16	16 KiB	128 MiB	finite	4.2	374	34	319
loopback	frame	16	4 MiB	128 MiB	finite	59.9	547	115	153
loopback	window	16	256 KiB	8 MiB	finite	3.9	624	18	94
loopback	window	16	256 KiB	32 MiB	finite	7.4	618	24	69
loopback	follow	16	4 MiB	128 MiB	follow	24.4	595	86	222
bufferbloat	default	16	256 KiB	128 MiB	finite	4.0	20	468	819
bufferbloat	streams	1	256 KiB	128 MiB	finite	1.0	15	54	190
bufferbloat	streams	256	256 KiB	128 MiB	finite	15.9	22	1519	2898
bufferbloat	frame	16	16 KiB	128 MiB	finite	0.3	20	501	811
bufferbloat	frame	16	4 MiB	128 MiB	finite	58.6	21	2840	2991
bufferbloat	window	16	256 KiB	8 MiB	finite	3.2	19	592	829
bufferbloat	window	16	256 KiB	32 MiB	finite	4.0	20	639	811
bufferbloat	follow	16	4 MiB	128 MiB	follow	16.0	22	1527	2940

An 8 MiB window loses nothing against 128 MiB at 256 KiB frames on either path. The peak stays under 16 MiB, the stock sender's send window, except with 4 MiB frames on finite stream windows, where every stream holds most of a frame at once.
follow with 4 MiB frames peaks at 16-24 MiB against 59-60 MiB for finite, and finishes sooner: a stream that can take its whole frame completes and releases it instead of waiting for credit.
Through the bottleneck every multi-stream cell runs at 19-22 MiB/s, near the 200 Mbit/s link.
A finding while pipelining: with four 4 MiB groups in flight per track, newest-first delivery starves each track's older groups while newer ones arrive, until their remainders fill the 128 MiB window and the oldest is evicted. That is eviction doing its job (the receiver drops the stalest data), but it means a deep pipeline of large frames needs a window of about tracks × in-flight × frame. The harness now caps each track at 1 MiB in flight.

moq-quic criterion, ThreadCpu:

receive_lease: leasing costs about 500 ns per stream over a plain read: 2.3x at 256 streams × 2 packets, +22% at 16 packets, where the per-packet work dominates. Worth trimming later; it is not on a per-byte path.
receive_evict: one stuck pass over N streams with partial frames takes ~170 µs at 16, ~130-215 µs at 256, ~510 µs at 1024 and 3.5-6.4 ms at 4096, so the cost per stream rises mildly past 1024 (cache misses, not a quadratic scan).

The earlier head's comparison against main, run back to back on the same machine (positive is slower):

Bench	Range	Notes
uring `session_lite` (native lease path)	-2% to -54%	`oneshot` -54%, `all-on` -11%, `all-off` -6%
moq-net `memory` (Buffered)	-2% to -23%	1 MiB × 128 streams -23%
moq-quic `receive_stop`	-1% to -38%
moq-quic `receive_loss` and `receive_reset`	mostly flat	one +13% outlier at 1024 resets
moq-net `frame_recv` (Buffered, 1200-byte chunks)	+1% to +15% typical	one +33% outlier, `upfront` 256 KiB × 64

The frame_recv slowdown is the per-chunk lease overhead (a VecDeque push and pop) on 1200-byte chunks.

Decisions

✅ The connection default is 128 MiB rather than 64 MiB. MIN_RECEIVE_WINDOW is the 64 MiB AnnounceInit cap plus 16 bytes of framing, so a 64 MiB default would be refused by its own check, and 128 MiB also holds two maximum-size (32 MiB) frames at once.
✅ The receive rank compares subscription priorities across broadcasts, as one i64. Ranking within broadcast fairness domains first belongs with Scope track priority, which owns those domains and changes priorities to i8. Proposed there as a follow-up rather than built here.
✅ Control streams outrank all media but stay eligible, as the quest decided: no protected class or reserve. A peer stuck on partial control requests makes its own media yield first; it cannot reach another peer's connection.
✅ The impaired-lane drills set only the stream window, since the connection window now has a 64 MiB floor. MAX_DATA pressure is covered by the receive-credit tests, whose raw-driver receivers take windows below the floor.
✅ STOP_SENDING is lazy: eviction stops the stream's reads locally, and the code goes out when the reader calls stop. The core does not choose a code. Mixing a lease and plain reads on one stream is unsupported.
✅ Count evictions on CreditStats (streams only), rather than relying on moq-net's Evicted errors, which non-moq-net readers never see.
✅ Rank by the subscription's group order, not always oldest first.
✅ Delete the m1 QUIC epic with its last child. Its per-broadcast fairness work lives in Scope track priority.
✅ Take over the PR now, despite the other session's recent push.
✅ Lease is a bytes::Buf plus release(), rather than keeping take/consume(n) or a guard that releases on drop.
✅ In this PR: criterion lease benches, the window bench rework, and the eviction metric on Stats.
✅ The shared receive buffer unifies with the cache Pool as an m2 quest: one budget per worker (the cache Pool splits per worker too), under pressure stop granting credit (never revoke it, fair share, no cross-peer eviction), and accounting first (QUICHE-style 8 KiB blocks only if a benchmark shows allocator or fragmentation cost).
✅ The receive rank keeps comparing raw group sequences across subscriptions of equal priority (option A, maintainer). Scoping the rank by subscription or broadcast domain stays with the existing follow-up, Scope track priority. Rejected: (B) packing a subscription identity between priority and sequence, which costs sequence bits and needs a stable short id; (C) widening the rank past one i64, which changes the transport trait.

Follow-ups

One receive budget per worker, shared with the cache.
Export streams_evicted from the relay's /metrics, which needs a per-node aggregate, and mirror it through moq-ffi and the bindings.
Trim the ~500 ns per-stream lease overhead receive_lease shows.
Rank received groups within broadcast fairness domains, with Scope track priority.
Re-rank a receive stream when its subscription's priority or order changes. Today the rank is set once, when the header decodes.
Group recovery refetches after any transport error, so an error that repeats on every copy loops. Eviction reports Evicted, which recovery treats as a verdict, so this PR no longer triggers it.

Checks

On the final head, 0c299dc:

just test interop --all passes.
just check passes except nix flake check, whose github-runner VM build needs user namespaces this host lacks; the stages after it pass when run directly.
just rs test -p moq-quic -p moq-net -p moq-tokio -p moq-relay passes (4571 tests) except drain::a_session_ignoring_the_goaway_is_closed_at_the_deadline and shutdown_signal::an_embedder_owns_the_signals, which fail the same way on main and are being fixed separately.

Earlier, on 142ffcc: the full Rust suite (just rs test --workspace --no-fail-fast) passed except export::redundant_importers_switch_through_bufferbloat_on_lite_07, a known main flake (redundant-failover-drift); cargo hack check -p moq-tokio --each-feature --exclude-features _transport --features aws-lc-rs and just test receive-credit (both Bun arms) passed. Not rerun since 2026-10-21: benchmark_receive_windows and the receive_lease/receive_evict benches.

🤖 Generated with Claude Code