Skip to content

fix(net): never leave a torn request message on the stream - #6433

Merged
kixelated merged 6 commits into
mainfrom
quest/m1/test-flakes-2/auth-union-renewal
Oct 24, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m1/test-flakes-2/auth-union-renewal

Conversation

@kixelated

@kixelated kixelated commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m1/test-flakes-2/auth-union-renewal.md.

Root cause

auth.test.ts's "a request waits for its token and survives union renewal" failed with "malformed request stream message". I reproduced it on the parent of moq-dev/moq#6255: 4 failures in 3,600 loaded runs, all on moqt-17/22. With extra logging I found the torn message. The publisher's REQUEST_OK answering the subscriber's update was cut short. The subscriber's update itself went out whole. The publisher wrote the type and the body in two writes. Writer.write can hand control back to the host after any write: the shared write budget yields once 4 ms of CPU time pass, which happens more often under load. The subscriber's FIN landed in that gap, so the publisher FINed after a bare type and the subscriber closed the session.

moq-dev/moq#6255 (merged today, after this quest was written) already makes every JS IETF/lite message a single write. I found no failures in 10,800 loaded runs on main. This PR adds a test that pins that fix and covers the related problems below.

The logged priority=255 was real but harmless. Before moq-dev/moq#6142, 255 was the wire value for priority 0. The subscriber sent it as a defaults-only update when its last subscriber left: maxAge fell back to MAX_SAFE_INTEGER, which differed from what had been sent. Rust and the JS lite subscriber never send that update. The JS IETF subscriber now matches them.

Changes

JS (js/net):

  • ietf/subscriber.ts: with no subscriber left, the update loop sends nothing (as on lite and in Rust).
  • ietf/subscriber.ts: an update that fails to encode now closes the track with the error and logs a warning, instead of being silently dropped. Encoding happens before the write (new internal Message.frame), so only a failed write still means "the publisher stopped reading".
  • ietf/subscriber.ts: on drafts 14 and 15 that failure also closes the session with an internal error. The update already spent a request ID, and those drafts require the next ID, so every later request would stall behind it. Rust's legacy header path does the same (a_failed_legacy_header_closes_the_session). Drafts 16+ accept a skipped ID, so they keep the session.
  • subscribe_priority.test.ts: three new cases, each run on every IETF ALPN (drafts 14 to 22, which also covers the draft 14-16 SubscribeUpdate encoding):
    • With every Writer.write yielding a host task (the worst case of the write budget) and a mocked clock, the last subscribers leave while an update is being answered. The session must stay up. On the parent of fix: m0 interop audit #6255 this fails 6/6 on drafts 17+ every run; it passes on this branch.
    • The last subscriber leaving sends no update. Fails 9/9 without the guard.
    • An update that fails to encode ends the track with its error. The session stays up on drafts 16+ and closes with an internal error on drafts 14/15. Times out 9/9 without the change, and 2/9 (drafts 14/15) without the session close.

Rust (rs/moq-net): the same split write existed in about 40 IETF control-message call sites: varint(T::ID).await? then encode(&msg), or buffer_varint(T::ID)? then buffer(&msg). It never tore a message in practice, because every error path resets the stream. But a failed encode, or a write future dropped between the two calls, left a bare type ID for the next write or FIN to complete. Some places could reach that:

  • TrackStatusOk's write is dropped mid-way on a grant revocation.
  • PublishDone ignores its errors, then closes the stream.

Writer::buffer_message(id, &msg) now buffers the type and the body together, all or nothing, and encode_message is built on it. Every production site uses one of the two. writer.rs's partial-encode test covers the type too. Control-byte stats still count the body without its type selector, as main's stats require; the adapter's stats test now writes through encode_message, so it pins that.

a_failed_legacy_header_closes_the_session used to force its header to fail through the split: with a 0-byte stream buffer, the body waited behind the type while the peer stopped the stream. A whole message can no longer fail that way on a fresh in-memory stream. The header now fails because its token is too big for the u16 length, after the stream opened and the request ID was spent. The assertions are unchanged: drafts 14/15 close the session, and draft 19 offers the namespace again.

The union renewal test still runs only on drafts 17 and 22. Drafts 14-16 have no grant, so the scenario does not exist there. The new tests cover the legacy encoding instead.

Public API / wire

  • Public API: none. Message.frame is internal to js/net/src/ietf, and Writer is crate-private in Rust.
  • Wire format: none. The JS IETF subscriber no longer sends a REQUEST_UPDATE/SUBSCRIBE_UPDATE of defaults just before tearing down a subscription whose last subscriber left.

Checks

  • just check passes after merging main.
  • The union renewal test ran 3,600 times under load (12 in parallel) on this branch, plus 10,800 on main: no failures.
  • The new torn-message test fails 6/6 every run against the code before fix: m0 interop audit #6255, and passes here.
  • Not run: just test interop --all. The wire format did not change. The only behavior change on the wire is one redundant update the JS subscriber no longer sends.

Decisions

Settled while iterating, on the recommended option:

  • A draft 14/15 update that fails to encode after spending its request ID:
    • ✅ Close the session with an internal error, as Rust does for a failed legacy header (recommended: the ID can never go out, and the peer requires it next).
    • Encode with a placeholder ID before allocating, to fail before spending one (a second encode per update, for a local bug).
    • Close only the track (leaves every later request stalled).

Follow-ups (not in this PR)

  • Writer.write's write budget can still yield between the multiple writes of a data-stream object header and its payload. Those streams are reset rather than FINed on teardown, so this was left alone. Worth an audit if a truncated object ever shows up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EeyiEy5FWz3z44DYZVo7vU

kixelated and others added 2 commits October 22, 2026 18:49
The union renewal flake was the publisher writing REQUEST_OK's type and body in
two writes: the shared write budget can yield to the host after any write, and
under load the subscriber's FIN landed in that gap, so the publisher FINed
after a bare type and the subscriber closed the session as malformed. #6255
already made every JS message one write; this pins that with a test that
yields after every write, and closes the rest:

- JS: the IETF subscriber no longer sends an update of defaults when its last
  subscriber leaves (the logged priority=255 was that), matching lite and
  Rust, and an update that fails to encode ends the track loudly instead of
  being swallowed.
- Rust: every IETF control message buffers its type with its body, all or
  nothing, via Writer::buffer_message / encode_message, so a failed encode or
  a cancelled write can no longer strand a bare type for the next write or a
  FIN to complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeyiEy5FWz3z44DYZVo7vU
@kixelated
kixelated force-pushed the quest/m1/test-flakes-2/auth-union-renewal branch from e6230ef to 9894b11 Compare October 23, 2026 01:59
@kixelated kixelated changed the title A union renewal never sends a torn request update fix(net): never leave a torn request message on the stream Oct 23, 2026
@kixelated
kixelated marked this pull request as ready for review October 23, 2026 02:00
@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: I reproduced the flake on the parent of #6255: the publisher's REQUEST_OK was written as a type and then a body, and the write budget's yield let the subscriber's FIN land between the two. On main, #6255 already writes each message whole. This PR pins that with a test that yields after every write, on every IETF draft. It also stops the JS subscriber's update of defaults when its last subscriber leaves (the logged priority=255), makes an update that fails to encode end the track instead of being swallowed, and makes every Rust IETF control message buffer its type and body together. just check passes. Interop was not run because the wire format did not change.

(written by Claude Opus 5.5)

@coderabbitai

coderabbitai Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 40 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4acb869b-12fc-48e4-8109-077e95efbc97
Commits

Reviewing files that changed from the base of the PR and between e61cee5 and c2ecd18.

Files selected for processing: 12
  • js/net/src/ietf/message.ts
  • js/net/src/ietf/subscriber.ts
  • js/net/src/subscribe_priority.test.ts
  • quest/m1/test-flakes-2/README.md
  • quest/m1/test-flakes-2/auth-union-renewal.md
  • rs/moq-net/src/coding/writer.rs
  • rs/moq-net/src/ietf/adapter.rs
  • rs/moq-net/src/ietf/publisher.rs
  • rs/moq-net/src/ietf/publisher/tracks.rs
  • rs/moq-net/src/ietf/request_stream.rs
  • rs/moq-net/src/ietf/subscriber.rs
  • rs/moq-net/src/ietf/subscriber/tracks.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-24T05:59:23.898401Z c2ecd18 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9894b11179

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread js/net/src/ietf/subscriber.ts
kixelated and others added 3 commits October 23, 2026 17:23
buffer_message now buffers the type and body separately so control stats
count the body without its type selector, as main's stats require.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drafts 14 and 15 require the next request ID, so an update that spent one
and never sent it would stall every later request. As in Rust's legacy
header path, that now closes the session with an internal error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated added this pull request to the merge queue Oct 24, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 24, 2026
@kixelated
kixelated enabled auto-merge October 24, 2026 05:57
@kixelated
kixelated added this pull request to the merge queue Oct 24, 2026
Merged via the queue into main with commit 02eba4a Oct 24, 2026
10 checks passed
@kixelated
kixelated deleted the quest/m1/test-flakes-2/auth-union-renewal branch October 24, 2026 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant