Skip to content

fix(net): drain received group tails after session close - #5762

Merged
kixelated merged 16 commits into
mainfrom
quest/m1/close-tail-drain
Oct 22, 2026
Merged

kixelated merged 16 commits into
mainfrom
quest/m1/close-tail-drain

Conversation

@kixelated

@kixelated kixelated commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Subscribers drop final groups their transport already delivered when a peer closes the session. This happens on released moq-lite versions and on both IETF session paths.

Approach

Close the transport first, then run a receive drain bounded by buffered data. Subscription state stays alive while accepted and queued group streams finish. IETF waits on control responses that can no longer arrive are settled. Both WebTransport adapters (moq-tokio and moq-uring) hand out buffered uni streams before they report the connection closed.

On lite with the ietf extension offered, a GROUP stream waits for the peer's SETUP before reading its header. Once the transport is closed and every remaining uni stream waits on a SETUP that is still unknown (truncated or never opened), nothing can deliver it, so the drain ends and those groups are cancelled as before.

The drain needs every transport to end an unfinished read once the session closes. qmux 0.6.3 and 0.6.4 did not: after teardown, Session handles kept the shared Streams table alive, and it still held each receive stream's channel senders, so a read past its buffered data stayed pending forever. That was fixed at the source in moq-dev/web-transport#437, released as qmux 0.6.5, and this PR bumps the workspace qmux requirement to 0.6.5.

Impact

  • No public API or wire format changes.
  • Dependency: qmux 0.6.3 -> 0.6.5. qmux 0.6.4+ uses trait API added in web-transport-trait 0.5.1 while declaring "0.5", so Cargo.lock also moves web-transport-trait 0.5.0 -> 0.5.2.
  • The production memory transport keeps its FIN acknowledgement semantics. The regressions use the test transport's delivery-ACK knob.

Decisions

Taken in this round

  1. How to fix the qmux teardown hang:
    • ✅ Targeted fix at the source in qmux (moq-dev/web-transport#437), consumed through the qmux 0.6.5 patch release and a version bump (maintainer, 2026-10-21)
    • Wait for the XL in-tree qmux (quest/m1/quic-qmux.md)
    • An interim guard in moq-tokio's async-backend adapter that ends blocked reads once closed() resolves (works around the dependency bug)
    • A drain that runs only on transports whose reads end after close (needs a new capability flag)

Taken in earlier rounds

  1. Post-close errors from the test transport carry the session's first close code. ✅ Carry the first close code, as a real transport reports the code the session ended with.
  2. The tail regressions wait until both ends hold the subscription. ✅ Wait for the subscriber's subscribe() to resolve before writing the tail. The control-response race is a known gap (see Follow-ups).

Validation

  • Merged main again (clean).
  • drain_ends_when_groups_wait_on_an_undeliverable_setup times out without the lite SETUP fix and passes with it; moq-net passes (2874 tests) and clippy is clean.
  • moq-gst with qmux 0.6.5: 5 of 5 full runs pass (199/199). Before the bump, a_closed_publisher_session_resumes_on_the_next_start and a_moqsink_restart_keeps_pads_when_caps_arrive_at_different_times timed out on every run.
  • just check origin/main: everything up to the flake step passes (Rust lint, docs, clippy, quest). nix flake check stops at the GitHub runner VM image, which needs user namespaces this sandbox lacks; unrelated to this diff. The steps after it were run by hand: ts-paths, shell, toml, markdown, gh pass, and workspace test-changed (the lockfile change selects every crate) passes 9440 + 516 + 224 tests. The nix and justfile format checks were not run (no such files changed).
  • just test interop --all was not rerun (no wire change); it passed on an earlier head.

Follow-ups

  • Drafts 14-16 drain only uni streams. A group whose SUBSCRIBE_OK is still unread on the control stream at close is settled, not delivered. If that matters, a quest could drain the control stream too (bounded by buffered data on a closed transport).
  • Upstream: qmux should require web-transport-trait = "0.5.1"; a consumer whose lockfile pins trait 0.5.0 fails to build qmux 0.6.4+.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 5 commits October 16, 2026 11:05
Retain subscription state while the closed transport drains buffered streams and settle IETF control waits. Deliver pending WebTransport uni streams before surfacing connection close. Regress published Lite versions and both IETF session paths, including control-first polling.

Co-Authored-By: GPT-6 <noreply@openai.com>
The optional production memory-transport acknowledgement change regressed the high-stream-count benchmark, so retain its existing semantics and use the test transport delivery-ACK knob for receiver-close coverage.

Co-Authored-By: GPT-6 <noreply@openai.com>
The existing TCP publisher restart regression passes on main but hangs with receive draining. A direct qmux 0.6.3 reproduction confirms unfinished stream reads remain pending after session and accept-queue closure while Session handles retain the shared stream table. Record the upstream dependency blocker and clarify that Session::closed does not finish receive draining.

Co-Authored-By: GPT-6 <noreply@openai.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Blocked on qmux 0.6.3 teardown. A direct TCP reproduction keeps a Session handle alive, waits for session and accept-queue closure, drains received data, and observes an unfinished stream read remain Pending. The shared stream table retains its channel senders. This drain exposes that defect: the existing GStreamer TCP publisher-restart test passes on unchanged main and reproducibly fails here waiting for source withdrawal.

just fix origin/main, quest check, and just test interop --all pass. The full gate reached 5,410 passes before that failure; 1,461 tests were not run after fail-fast. Three protocol and both real WebTransport regressions fail without the corresponding fixes and pass with them. The optional production memory ACK change is omitted.

The quest remains open. Fix the qmux lifecycle upstream, coordinate with the existing close/credit work, then release/pin and rerun the gate. No timeout workaround, merge, deployment, or release was made.

(written by GPT-6)

@kixelated

Copy link
Copy Markdown
Collaborator Author

The terminal stream hang now has a focused source fix in moq-dev/web-transport#437. The shared stream table retained channel senders after backend exit, so draining an unfinished receive stream never completed while a Session handle remained alive. The fix releases those senders and guards concurrent registration, preserving buffered bytes and received FINs without changing the public API or wire format.

Source validation passes on clean 601e7fb: pinned-Nix just fix, just check, and just test, including real TCP uni/bidi loopback on QMux01/QMux02 and Rust/TypeScript interop. The exact GStreamer test that timed out here with published qmux 0.6.3 passes in 2.565s using a temporary source override in an isolated checkout. This PR's head, config, and lockfile are unchanged.

Remaining blocker: review/merge the upstream fix, publish a qmux release containing it, then update the dependency and lockfile here and rerun full clean gates and interop. No crate publication or unpublished dependency pin was added.

(written by GPT-6)

kixelated and others added 2 commits October 20, 2026 13:12
…rain

# Conflicts:
#	rs/moq-net/src/ietf/session.rs
#	rs/moq-net/src/lite/session.rs
#	rs/moq-net/src/lite/test_transport.rs
A post-close transport error now carries the session's first close code, so a
driver that ends on that error reports the same code the session closed with.
The tail regressions wait until both ends hold the subscription, so they test
the group tail rather than a SUBSCRIBE_OK still in flight. Record that main
still pins qmux 0.6.3 and that the fix must come from moq's own stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
qmux 0.6.5 (moq-dev/web-transport#437) ends receive-stream reads at
session teardown, which the close-tail drain relies on. It uses trait
API from web-transport-trait 0.5.1, so the lockfile moves that too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review October 22, 2026 02:01
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 11 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 42533117-d0ed-4460-9368-b24e40936ffc
📥 Commits

Reviewing files that changed from the base of the PR and between 1ff226c and f2e52db.

⛔ Files ignored due to path filters: 1
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing: 15
  • Cargo.toml
  • quest/m1/README.md
  • quest/m1/close-tail-drain.md
  • rs/moq-net/src/ietf/adapter.rs
  • rs/moq-net/src/ietf/session.rs
  • rs/moq-net/src/ietf/subscriber.rs
  • rs/moq-net/src/lite/session.rs
  • rs/moq-net/src/lite/subscriber.rs
  • rs/moq-net/src/lite/test_transport.rs
  • rs/moq-net/src/session.rs
  • rs/moq-net/tests/integration/session_close.rs
  • rs/moq-net/tests/integration/support/harness.rs
  • rs/moq-tokio/src/quic/web/session.rs
  • rs/moq-uring/src/quic/web.rs
  • rs/moq-uring/tests/integration/web.rs

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 16ff3d34-7c2b-4da3-b66a-7b06521ec5d8
📥 Commits

Reviewing files that changed from the base of the PR and between 4329029 and 1ff226c.

⛔ Files ignored due to path filters: 1
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing: 5
  • quest/m1/README.md
  • quest/m1/close-tail-drain.md
  • rs/moq-net/src/ietf/session.rs
  • rs/moq-net/src/ietf/subscriber.rs
  • rs/moq-net/src/lite/session.rs
💤 Files with no reviewable changes: 2
  • quest/m1/README.md
  • quest/m1/close-tail-drain.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.

Walkthrough

The changes update qmux and revise IETF and Lite session handling so accepted streams can drain after session closure. IETF receive adapters preserve real transport errors. WebTransport accept paths process buffered streams before returning close errors. New tests cover buffered unidirectional streams and group tails across peer close, including cases where the receiver polls later.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 1ff22

Buffered group streams can drain after session close. No demonstrated issue remains that should block merging, though the qmux fix mapping was not independently verified.

Pre-merge checks: Passed 4 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Docstring Coverage Inconclusive Docstring coverage is 31.08% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 74 functions across 11 files. (1 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the primary change: draining received group tails after session close.
Description check Passed The description is directly related to the changeset and explains the problem, implementation, dependency update, validation, and known follow-ups.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 31.08% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 74 functions across 11 files. (1 skipped: 1 too large.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches
✨ Simplify code
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-22T05:01:51.907603Z f2e52db New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fbed668b98

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-net/src/lite/subscriber.rs Outdated
Comment on lines +644 to +645
let closed = self.uni.poll(waiter).is_ready();
let drained = self.uni.children.poll(waiter).is_ready();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Settle missing SETUP before waiting for lite children

When lite-07 offers the IETF extension and a peer opens a GROUP stream but closes before completing its SETUP stream, GroupRecv remains pending in peer_setup.poll_ietf(): the failed SETUP child exits without filling that slot, while this drain waits for every child to finish. Transport closure cannot resolve the shared setup slot, so the spawned session driver and its subscription state remain retained indefinitely; settle or fail PeerSetup before waiting for the children.

AGENTS.md reference: AGENTS.md:L17-L18

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and it also hangs when the SETUP stream was never opened before close. Fixed in 4329029: GROUP streams parked on the peer SETUP are counted, and once the transport is closed, the SETUP is still unknown, and every remaining uni stream is parked, the drain ends (those groups are cancelled, as before the drain). Regression: drain_ends_when_groups_wait_on_an_undeliverable_setup covers both the truncated and the missing SETUP, and timed out without the fix.

(Written by Claude Opus 5.5)

With the ietf extension offered, a GROUP stream parks on the peer's SETUP
before reading its header. Once the transport is closed and every
remaining uni stream is parked that way, no stream can still deliver the
SETUP (it was truncated or never opened), so the drain waited forever.
Count parked streams and stop draining when they are all that is left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 4329029ac4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@kixelated
kixelated added this pull request to the merge queue Oct 22, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merged origin/main (no rebase) and queued at 1ff226ce60889808a2788702e0cf8fadfec49122.

  • The only conflict was quest/m1/close-tail-drain.md, where main had added a Required blocker on the in-tree qmux quest. The qmux 0.6.5 bump clears that blocker and this PR finishes the quest, so the merge deletes the quest file and its quest/m1/README.md entry. The code merged without conflicts.
  • Checks: just check origin/main passes up to nix flake check, which can't run in this sandbox (user-namespace limit). I ran the later steps by hand: ts-paths, markdown, shell, toml, and gh pass. Workspace tests: 9454/9455 pass; capture-test (516) and play (222) pass. The drain regressions pass (session_close::*, older_lite_close_keeps_ack_drain, drain_ends_when_groups_wait_on_an_undeliverable_setup).
  • Two moq-relay tests fail on both this branch and origin/main, 5 of 5 runs each, with the same assertion: drain::a_session_ignoring_the_goaway_is_closed_at_the_deadline ("drained before the deadline") and shutdown_signal::an_embedder_owns_the_signals ("relay exited after ~500ms, short of the 1s drain window"). This PR didn't cause them and doesn't make them worse.
  • moq-dev/moq#6231 also edits quest/m1/close-tail-drain.md and the qmux dependency. Whichever PR lands second has to reconcile them.

(Written by Claude Opus 5.5)

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 22, 2026
# Conflicts:
#	quest/m1/close-tail-drain.md
@kixelated
kixelated added this pull request to the merge queue Oct 22, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merged origin/main again in f2e52db to clear a quest-only conflict: main reworded a plan bullet in quest/m1/close-tail-drain.md, and this PR deletes that file since it finishes the quest, so it stays deleted. No other links to it remain. quest check, the Rust lints and clippy, the markdown/shell/toml/gh lints, and the changed-crate Rust tests pass locally; nix flake check can't run here (user-namespace limit). Codex was clean on 4329029 and the code is unchanged since, so this goes back into the queue.

(Written by Claude Opus 5.5)

Merged via the queue into main with commit 96a8c90 Oct 22, 2026
10 checks passed
@kixelated
kixelated deleted the quest/m1/close-tail-drain branch October 22, 2026 06:49
kixelated added a commit that referenced this pull request Oct 22, 2026
Main's #5762 added RecvError to keep a real receive stream's error
through the adapter, while this branch added StreamFailure for both
directions. Unify them as one generic AdapterError that keeps the
transport error typed for send and receive, and forwards close_kind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Oct 22, 2026
Keep #5762's RecvError over leased reads, forwarding close_kind so an eviction still surfaces as Evicted. Port its scripted-uni close test to poll_read_chunk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant