Skip to content

feat(quic): hold connection credit until the receiver releases it - #5675

Merged
kixelated merged 28 commits into
mainfrom
quest/m1/quic/receive-credit
Oct 16, 2026
Merged

kixelated merged 28 commits into
mainfrom
quest/m1/quic/receive-credit

Conversation

@kixelated

@kixelated kixelated commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Completes quest/m1/quic/receive-credit.md (deleted here) and unblocks quest/m1/quic/receive-memory.md, which gets a short note on the seam it should finalize.

What

Native receivers can read stream bytes without returning connection credit until they drop an owned Credit handle. Stream credit (MAX_STREAM_DATA) still advances on reads, so a stream window smaller than a message keeps it flowing while the connection window (MAX_DATA) stays spent.

moq-quic

  • Chunks::hold() -> Credit takes the connection credit for the bytes read so far; finalize then returns only stream credit.
  • Connection::release(Credit) -> ShouldTransmit returns it. Credit is a move-only token (len, is_empty, split, merge), so double release cannot be written. Each connection stamps its credit with a process-unique, crate-private ConnectionTag, and release and merge panic on a credit from another connection (an empty Credit::default() adopts the tag of the first credit merged in). Releasing more than the connection holds also panics.
  • ConnectionStats::credit: CreditStats { held, available }: bytes held by the application, and bytes the peer may still send (including credit not yet advertised). Together they separate held memory from bytes that can still arrive.
  • Every terminal path (FIN, STOP_SENDING, RESET_STREAM, stop-then-reset, final-size gaps, late data on a stopped stream, state reclamation) already returned only unread bytes. So held bytes account exactly once, whether they are released before or after their stream is reclaimed. Shrink debt applies to releases like any other credit.
  • MAX_DATA batching: an update is now also due when diff > 0 and the peer is within the batching threshold (window/8) of the last limit it was told. This is checked on release/read and on each arrival. Without it, a release below the threshold never goes out: held credit may never add up to window/8, and the receiver ignores DATA_BLOCKED. This applies to the plain read path too. The extra frames only appear when the receiver is the bottleneck.

moq-tokio driver / moq-uring driver (same shape in both)

  • RecvStream::poll_read_held(cx, dst) -> Poll<Result<Option<Credit>, _>> is the copying read moq-net uses, except it yields the driver's Credit for exactly the bytes copied into dst (credit.len() is the count), or None at FIN. tokio also has async read_held. Plain reads are unchanged and return credit as before.
  • The driver Credit wraps moq_quic::Credit with its connection: len, is_empty, split(n), merge(other). Dropping it releases it, even after the stream or connection is gone. merge panics across connections. So over-release, release to the wrong connection, and leaks are all unwritable (short of mem::forget), and the moq-quic panic is unreachable from the drivers.
  • moq-uring: poll_closed's read-ahead keeps its credit until the application reads those bytes, as if they were still in moq-quic's reassembly buffer. A plain read returns it as it drains the backlog; a held read moves it into the returned Credit; stop and drop return the rest.
  • moq-uring Connection::stats() -> moq_quic::ConnectionStats, matching the tokio driver and quinn; both drivers read credit as conn.stats().credit, which counts outstanding driver Credits and io_uring read-ahead as held. On io_uring this inherent stats() shadows the poll::Session::stats trait method on concrete calls (call the trait by path for the web_transport_trait::Stats shape); tokio avoids that because its driver and WebTransport session are separate types.

No callbacks, no sender-side shedding, no victim selection. Those belong to receive-memory.

The seam receive-memory builds on

A reader buffer keeps one merged Credit next to its bytes: each poll_read_held appends to the buffer and merges the credit. When a complete unit of n bytes is parsed and handed off or discarded, drop(credit.split(n)); read-ahead stays in the remainder and is attributed to the next unit. Cancellation, errors, and teardown release by dropping the buffer. What is left to decide there is how moq-net's transport::poll::RecvStream carries it (likely an associated Credit type with a no-op for browser/iroh/WebTransport).

Settled decisions

  1. Driver API shape:
    • hold() plus release(bytes) by count
    • ✅ owned driver Credit handles returned by each held read (poll_read_held -> Option<Credit>), with split/merge covering partial consumption. Over-release cannot be written.
  2. Dropping a Credit:
    • ✅ releases it (RAII). Rust cannot make a drop impossible to forget, and the consumer has many cancel/error paths where a leaked handle would permanently shrink the window into a silent livelock.
    • #[must_use] token that leaks on drop
  3. Driver Credit:
    • ✅ a thin driver wrapper that carries the connection, so it can release itself and refuse cross-connection merges
    • re-export moq_quic::Credit and release through Connection
  4. io_uring poll_closed read-ahead (new, recommended and taken by the agent):
    • ✅ always keeps its credit until the application reads it, on every stream. This removes the stream-level hold() mode and its unheld bookkeeping, and matches how unread bytes in moq-quic already behave. Plain streams now advertise read-ahead bytes' MAX_DATA when the application reads them rather than when poll_closed reads ahead (at most 64 KiB per watched stream).
    • hold read-ahead only on streams opted in with a hold() mode
  5. ✅ Keep the new MAX_DATA rule on ordinary reads.
  6. ✅ Keep the panic on over-release inside moq-quic (now unreachable from the drivers), and also panic on a credit released into or merged with another connection (Codex P1). The tag is crate-private so moq_quic::Checkpoint (feat(quic)!: reliable reset from a send-stream checkpoint #5690) can reuse it.
  7. ✅ Delete the quest file here; the seam note lives in receive-memory's plan.
  8. ✅ Keep poll_read_held separate for now; receive-memory folds it into poll_read.
  9. Stats accessor naming next to the Credit handle:
    • ✅ drop io_uring's Connection::credit() for Connection::stats() -> moq_quic::ConnectionStats, matching the tokio driver and quinn; read credit as stats().credit in both drivers
    • rename it credit_stats()
    • keep credit()

Public API / wire impact

  • API (additive; provisional until receive-memory consumes it):
    • moq_quic::{Credit, CreditStats}, Chunks::hold, Connection::release, ConnectionStats::credit.
    • moq_tokio::quic::driver::Credit (len, is_empty, split, merge, release on drop), moq_tokio::quic::driver::RecvStream::{read_held, poll_read_held}.
    • moq_uring::quic::Credit (same methods), moq_uring::quic::RecvStream::poll_read_held, moq_uring::quic::Connection::stats.
    • No hold/release on the driver RecvStreams.
  • Not plumbed: moq-net's transport::poll::RecvStream trait and the WebTransport wrappers (moq_tokio::quic::web, moq_uring::quic::web). The consumer decides that shape.
  • Wire: no format change. Peer-visible differences: extra MAX_DATA frames when a peer is close to its limit, and on io_uring, read-ahead bytes' MAX_DATA waits until the application reads them. No MoQ message or negotiation changes. Browser and iroh are untouched.

Tests

  • moq-quic state tests: held_credit_returns_once (FIN / stop / reset / stop-then-reset with a gap and late data, release after reclamation), held_credit_pays_shrink_debt, release_from_another_connection (panics even when the other connection holds enough to absorb it; fails without the tag check), merge_from_another_connection (panics), default_credit_adopts_its_connection.
  • moq-quic simulator: held_credit_spends_connection_window_only. A 1000 B stream window delivers a 4000 B connection window while it is held. A 100 B release (threshold 500) unblocks the peer, and so does one queued while the peer still had room, with no read in between. Both halves fail without the new MAX_DATA rule.
  • moq-tokio driver: held_reads_spend_the_connection_window (real sockets): held reads spend the window, dropping a 100 B split unblocks, and credit outlives its stream until dropped.
  • moq-uring tests/credit.rs: the same flow, plus read-ahead keeping its credit, a plain read returning it, a held read taking it, stop returning only the read-ahead, and credit outliving the stream. Passed under MOQ_STRICT=1.
  • Benchmark benchmark_receive_credit (ignored; in the Rust nightly group) sweeps 1/64/4096 streams × 1200 B/64 KiB reads. Held and plain reads cost the same within noise (43-73 ns per read with the connection tag, on a loaded shared box), flat across stream counts. It measures moq-quic; the driver Credit adds one Arc/Rc clone per held read and one connection lock per non-empty drop, which release(bytes) also took.
  • just check: all 4695 tests pass after merging the latest main, lint clean.
  • Skipped just test interop --all: there is no wire, ffi, or gateway change.

Not covered

  • Composing with reliable reset (RESET_STREAM_AT) waits until that quest lands. A reliable prefix would be held like any other read; its final-size gap goes through the existing reset accounting.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 10 commits October 15, 2026 19:35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…batim

Unmodified copies, not yet compiled, so a reviewer can diff the adaptation
that follows against upstream:

- quinn-rs/quinn 7616e6b2782722f3ce4a1b181ef08817d7f545e4 quinn/src ->
  rs/moq-tokio/src/quic/driver (lib.rs -> mod.rs), the commit moq-quic
  forked from.
- moq-dev/noq e27c0ce00741d81c8620b98e345d4f2c88e0768c web-transport-moq/src ->
  rs/moq-tokio/src/quic/web (lib.rs -> mod.rs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop every moq-noq dependency. moq-tokio's QUIC backend now drives moq-quic
through quinn's async layer (`quic::driver`) with web-transport-moq folded in
as `quic::web`; moq-uring's sans-IO driver moves from moq-noq-proto to
moq-quic. Upstream noq stays only behind moq-tokio's `iroh` feature.

- The `noq` cargo feature is `quic` everywhere it was forwarded.
- `moq_tokio::noq::Error` is `moq_tokio::quic::Error`; `Error::Noq` is
  `Error::Quic`.
- moq-quic takes `moq_clock::Instant` and raises a once-per-connection
  Draining endpoint event, which backs the driver's `wait_all_draining`.
- qlog writes one trace per endpoint (per dial on moq-uring's connect side),
  since moq-quic holds one sink per transport config.
- Session stats report the congestion controller's bandwidth estimate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The capsule reader maps a lost stream to UnexpectedEnd, so every ordinary
session close warned "failed to read capsule". A read error while the
connection is closed is now debug; one on a live connection still warns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Relay peak RSS on the fanout workloads matches moq-noq on origin/main, which
was the last open part of the Goal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A moq-quic read can now keep its connection flow control credit:
`Chunks::hold` takes it as a `Credit`, and `Connection::release` returns
it. Stream credit still follows reads, so a finite stream window never
forces a whole message to fit. Every terminal path already returned only
unread bytes, so held bytes account once whether they are released
before or after the stream is reclaimed.

Credit too small to clear the usual MAX_DATA batching threshold is
announced once the peer is within that threshold of its limit, on
release or on the next arrival. Held credit may never add up to the
threshold, and the receiver ignores DATA_BLOCKED, so without this a
small release could leave the peer blocked indefinitely.

The tokio and io_uring drivers opt in per receive stream with `hold`
and return bytes with `release`; dropping the stream releases the rest.
io_uring's read-ahead is not releasable by the application and returns
on stop. `ConnectionStats::credit` reports held and still-available
credit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Quest outcome: the receive-credit seam is in, the tests pass, and the PR stays a draft stacked on #5664.

Open decisions for the maintainer, each with a recommendation:

  1. MAX_DATA rule on plain reads. The "peer within window/8 of its limit" trigger applies to every read, not just to held releases, so upstream quinn batching changes slightly. Recommend keeping it: the rule depends only on connection state, and the extra frames only show up when the receiver is the bottleneck.
  2. Adapter shape. hold() plus release(bytes) by count, rather than an owned RAII credit handle per read. A count fits moq-net's copying poll_read and a future trait method with a no-op default. Recommend keeping it and finalizing the shape in receive-memory.
  3. Over-release panics instead of returning an error, since it is a caller bug and silently growing the window would defeat the memory bound. Recommend keeping the panic.
  4. Quest deleted here rather than kept until the consumer settles the API. That note now lives in receive-memory's plan. Recommend deleting.

Follow-ups:

  • Plumbing through moq-net's transport trait and the WebTransport wrappers is receive-memory's job; no new quest needed.
  • Reliable reset composition is already tracked in quest/m1/quic/reliable-reset.md.
  • The base branch has an unused timed test helper warning in rs/moq-tokio/src/client.rs (feat(quic)!: run moq-tokio and moq-uring on moq-quic #5664).

(Written by Claude Opus 5.5)

kixelated and others added 5 commits October 15, 2026 21:50
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Carry #5638's steering changes onto the renamed moq-quic drivers: the
shard CID comes from moq_sock::shard::cid, and moq-uring stops greasing
the QUIC bit on sharded endpoints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated and others added 9 commits October 15, 2026 23:01
Replace RecvStream::hold/release(bytes) in both drivers with a copying
poll_read_held that yields a driver Credit for exactly the bytes read.
Credit splits and merges, and dropping it returns the connection credit,
so over-release and cross-connection release cannot be written.

io_uring's poll_closed read-ahead now keeps its credit until the
application reads it: a plain read returns it, a held read takes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Endpoint::close now refuses attempts still queued for accept, and
wait_all_draining waits for accepts partway through Incoming::accept, so
a server shutdown cannot release the socket before those peers hear back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The server socket retried every failed recv, so a persistent error spun
the endpoint task while tokio kept readiness asserted. Only WouldBlock
retries now, matching the driver's tokio runtime.

Also cap moq_quic at info like noq was, fix the driver doc example, and
point the moq-uring README at the just recipe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Match the tokio driver and quinn: credit is read as stats().credit in both
drivers. The inherent stats() shadows the poll::Session trait's on concrete
calls, so the web test reads UDP bytes from it directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from quest/m1/quic/fork/switch to main October 16, 2026 16:04
# Conflicts:
#	quest/m1/quic/receive-credit.md
#	rs/moq-tokio/src/quic/driver/connection.rs
#	rs/moq-tokio/src/quic/driver/mod.rs
#	rs/moq-tokio/src/quic/driver/recv_stream.rs
#	rs/moq-tokio/src/quic/driver/tests.rs
#	rs/moq-uring/src/quic/driver/stream.rs
#	rs/moq-uring/src/quic/mod.rs
@kixelated
kixelated marked this pull request as ready for review October 16, 2026 17:18
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T17:57:52.176646Z cc58807 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 888523e47b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-quic/src/connection/streams/state.rs
@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 26 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5eb7f2b1-623f-4dde-acad-f7e20780d3ea
📥 Commits

Reviewing files that changed from the base of the PR and between d29e6ee and cc58807.

📒 Files selected for processing: 21
  • quest/m1/quic/README.md
  • quest/m1/quic/receive-credit.md
  • quest/m1/quic/receive-memory.md
  • rs/moq-quic/src/connection/mod.rs
  • rs/moq-quic/src/connection/stats.rs
  • rs/moq-quic/src/connection/streams/mod.rs
  • rs/moq-quic/src/connection/streams/recv.rs
  • rs/moq-quic/src/connection/streams/state.rs
  • rs/moq-quic/src/lib.rs
  • rs/moq-quic/src/tests/mod.rs
  • rs/moq-tokio/src/quic/driver/connection.rs
  • rs/moq-tokio/src/quic/driver/mod.rs
  • rs/moq-tokio/src/quic/driver/recv_stream.rs
  • rs/moq-tokio/src/quic/driver/tests.rs
  • rs/moq-uring/src/quic/driver/connection.rs
  • rs/moq-uring/src/quic/driver/mod.rs
  • rs/moq-uring/src/quic/driver/stream.rs
  • rs/moq-uring/src/quic/mod.rs
  • rs/moq-uring/tests/credit.rs
  • rs/moq-uring/tests/web.rs
  • sh/gh/nightly.sh
✨ Finishing Touches
✨ Simplify code
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

kixelated and others added 2 commits October 16, 2026 10:28
# Conflicts:
#	quest/m1/quic/README.md
#	quest/m1/quic/receive-credit.md
A Credit from one connection released into another succeeded whenever the
other held at least as many bytes, silently moving receive window between
them. Each StreamsState now carries a process-unique ConnectionTag that
Chunks::hold stamps on its Credit; release and merge panic on a foreign
one. An empty default Credit adopts the tag of the first credit merged in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: cc58807825

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging at cc58807825b4193ea340a986bda0af56667b2218 after maintainer approval, now that #5664 has landed.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 16, 2026
Merged via the queue into main with commit 1901af9 Oct 16, 2026
13 of 14 checks passed
@kixelated
kixelated deleted the quest/m1/quic/receive-credit branch October 16, 2026 19:04
kixelated added a commit that referenced this pull request Oct 16, 2026
Checkpoint now carries the connection tag #5675 added for Credit, next
to the streams' generation. A checkpoint from another connection panics,
as a foreign Credit does, even where that connection's stream of the
same id holds as much data, which the old offset assert let through.
A stale generation on the right connection, from rejected 0-RTT, still
fails with ClosedStream since that is a legitimate race.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Oct 16, 2026
Brings in receive credit (#5675). Its ConnectionTag now also tags the
send group and domain tokens, so moq-quic refuses a token from another
connection as its docs promised, instead of indexing whichever of its own
groups shares the slot. The drivers' pointer checks became redundant and
are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant