Skip to content

quest(m0): plan the Interop failures on main - #5388

Merged
kixelated merged 5 commits into
mainfrom
plan/interop-main-failures
Oct 13, 2026
Merged

kixelated merged 5 commits into
mainfrom
plan/interop-main-failures

Conversation

@kixelated

@kixelated kixelated commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Plans two m0 quests for the red Interop job on main. The scheduled run of 2026-10-12 (38284317921) and PRs that only touch test/interop, such as #5381, both fail.

Root causes

  1. unroutable in the cross-client matrix -> quest/m0/upgrade-unroutable.md
    • When WebSocket wins the connect race (common on the CI runner), feat(tokio)!: upgrade a WebSocket fallback session to QUIC #4180's upgrade re-announces the relay's routes over QUIC under new route generations. Since fix(net): refuse new requests from retired sources #5294 (f0fcbef, 2026-10-11), RequestValidity treats every epochless handle resolved before the upgrade as retired, so its next track request fails with Unroutable. Rust (moq export), Go, and C++ subscribers exit with that error. The C client stalls and times out, and the clients that failed never close their connections, so the relay idles them out.
    • Reproduced deterministically in websocket_upgrades_to_quic. A late subscribe through the pre-upgrade handle gets Unroutable, and forcing RequestValidity::check to pass makes it succeed. The announce consumer does get Restart, but only after the origin's 300 ms update_hold, while the handles retire at once. A request inside that window fails before any Restart could be followed (Codex caught the hold on review). A fresh routed_broadcast works. The local full matrix passes because QUIC wins locally.
  2. Released wire compatibility -> quest/m0/wire-compat-green.md
    • The latest release (moq-cli 0.14.3, moq-relay 0.17.3, hang 0.21.12, @moq/net 0.4.3) made every planned-breaks.json entry stale, so the resolver aborts before any cell runs.
    • compat.sh runs this checkout's js-native/subscribe.ts against released @moq/*. Since feat(js)!: data consumers return each value's timestamp #5099 that script calls latest() on a Json.Snapshot.Consumer, which released @moq/json 0.4.2 lacks. Every publisher=released fetch cell has failed since the 2026-10-11 nightly.

Decisions (maintainer, 2026-10-13)

What an upgrade means on an epochless route:

  1. An upgrade is a transport swap that continues the old route identity.
  2. ✅ An upgrade is a Restart. "No re-requests after Unroutable please, explicitly Restart only." Handles retire only once their Restart is delivered, and first-party consumers follow it and never retry after Unroutable.
  3. Never upgrade epochless sessions.

Which client the released JS lanes run:

  1. ✅ Their own observer in test/interop/compat/client.ts, using only API the released packages have.
  2. A client staged from the released package's tag.

Placement:

  1. ✅ Both quests stay in m0.
  2. Move wire-compat to m1.

Public API: none. Wire: none.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

The cross-client matrix fails with unroutable whenever WebSocket wins the
connect race: #4180's upgrade re-announces every route under a new
generation, and #5294 retires the epochless handles resolved before it.
Released wire-compat aborts on planned breaks the last release made stale,
and its released JS lanes run a client that uses a current-only API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 2 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b4203263-efe5-4b81-bbf3-416bc342382d
📥 Commits

Reviewing files that changed from the base of the PR and between ae3313c and 37d555d.

📒 Files selected for processing (4)
  • quest/m0/README.md
  • quest/m0/upgrade-unroutable.md
  • quest/m0/wire-compat-green.md
  • quest/m1/transport-upgrade/README.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-13T04:12:31.229295Z fafcf8e Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9986ded19

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread quest/m0/wire-compat-green.md Outdated
- The released JS lanes run this checkout's client. `compat.sh` copies
`test/interop/clients/js-native/subscribe.ts` into the released `@moq/*`
environment. #5099 (2026-10-10) changed that script to call
`catalog.latest()`, which released `@moq/hang` 0.5.2 lacks. Every

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Identify the package that actually owns latest()

When this quest is implemented, investigating @moq/hang will target the wrong release: subscribe.ts constructs catalog as a Json.Snapshot.Consumer, and latest() is defined by @moq/json in js/json/src/snapshot/consumer.ts. The compatibility resolver also pins @moq/json independently, so the plan should name the released @moq/json version that lacks this method; otherwise an implementer could remeasure or modify the wrong package.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. latest() is on @moq/json's Json.Snapshot.Consumer, and released @moq/json 0.4.2 lacks it. The quest and the PR body now name that package.

(Written by Claude Opus 5.5)

kixelated and others added 2 commits October 12, 2026 21:01
An epochless upgrade is an explicit Restart that first-party consumers
follow, never a retry after Unroutable. Planning found the upgrade retires
old handles without announcing that Restart. The released JS compat lanes
get their own observer, and both quests stay in m0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a4fce52a2b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread quest/m0/upgrade-unroutable.md Outdated
Comment on lines +82 to +85
followed. Find why the existing epochless-replacement Restart (the
`PendingUpdate::Restart` path in `model/origin.rs`) does not fire here.
The likely reason is that the replacement comes from the same `via`, so
it reads as an update in place. Fix it in the origin, not in moq-tokio, so

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Correct the same-via Restart diagnosis

The proposed cause is contradicted by the existing origin logic: TableCursor::update emits apply_restart whenever the selected route's Instance changes, without exempting the same via, and identical_reannounce_restarts_without_an_epoch already verifies that an identical epochless reannouncement produces Restart. Epochless restarts are instead held by the default 300 ms update_hold, so a reproduction that checks immediately can observe no event. Directing the implementation to change same-via handling risks modifying the wrong invariant; first determine whether the reported absence is the hold or another upgrade-specific path.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and confirmed locally. With a 2 s wait, the announce consumer gets Restart for cam after the upgrade. My first check had stopped inside the 300 ms update_hold. The quest now says the signal is late, not missing: RequestValidity retires the handle as soon as the new route wins, while its Restart waits out the hold. It points the fix at aligning the two in the origin and explicitly leaves same-via handling alone.

(Written by Claude Opus 5.5)

An epochless Restart waits out the origin's update_hold while the old
handles retire at once, so a request in that window gets Unroutable before
any Restart could be followed. Align retirement with the Restart instead
of changing same-via handling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: fafcf8e.

[P2] Preserve instance isolation while delaying notification — quest/m0/upgrade-unroutable.md:88–91 recommends keeping RequestValidity live and accepting new requests throughout the hold. A draining connection does not prove it still serves the old publisher: if epochless publisher A is replaced by B during upgrade, QUIC can announce B before WebSocket delivers its restart. Requests sent through the retained A handle carry only path/track and epoch=None (rs/moq-net/src/lite/subscriber.rs:4505–4509, 4798–4802), so the peer can answer with B's content. This reopens the cross-instance failure fixed by #5294. Keep invalidation immediate when old-instance service cannot be proven; align Restart delivery or park pending requests rather than universally allowing them through the hold. Add a controlled publisher-replacement-during-upgrade regression asserting that an old handle never receives B's bytes.

The corrected hold diagnosis, explicit Restart-only following, and isolated released-API client are sound directions. Static review of the plan and relevant source only; no tests run.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fafcf8e9f4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread quest/m0/upgrade-unroutable.md Outdated
Comment on lines +89 to +90
`RequestValidity` retires when its `Restart` is published, so the old
route keeps serving new requests through the hold. The old session is

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Tie retirement to Restart delivery, not publication

If “published” means queued in the origin, this recommendation describes the current behavior and leaves the failure intact: OriginState::sync_route calls TableCursor::update, which queues PendingUpdate::Restart, before routes.poke_below invalidates the Selection; the 300 ms delay is applied later by AnnounceConsumer::poll_next. Specify that retirement must wait until the held event becomes deliverable, or make this Restart unheld, so an implementation cannot repeat the existing ordering while claiming to complete the quest.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The quest now says retirement waits until the held Restart is deliverable, not merely queued, and names the current ordering (sync_route queues before poke_below; the hold is applied in AnnounceConsumer::poll_next).

(Written by Claude Opus 5.5)

Comment thread quest/m0/upgrade-unroutable.md Outdated
client's error path to close its session, as a separate commit. The C
client's `fail()` and the C++ client on a consumer error both look suspect.

Public API: an announce `Restart` on upgrade is new observable behavior.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Report the existing Restart behavior accurately

The reproduction above establishes that an upgrade already emits Restart after update_hold, so the event itself is not new observable behavior. The impact is the changed retirement ordering and automatic consumer following, plus any binding API that the audit actually adds; describing the already-existing event as new would make the implementation PR's required public-API accounting inaccurate.

AGENTS.md reference: AGENTS.md:L60-L60

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The Public API line now says the upgrade's Restart already exists; what changes is when old handles retire relative to it, and that consumers follow it.

(Written by Claude Opus 5.5)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge summary

This PR adds two m0 quests for the red Interop job on main.

Decisions (maintainer, 2026-10-13):

  • On an epochless route, an upgrade is an explicit Restart. Consumers follow Restart and never retry after Unroutable.
  • The released JS compat lanes get their own observer in test/interop/compat/client.ts.
  • Both quests stay in m0.

Reviews: Codex reviewed fafcf8e9. Its four findings were the @moq/json owner, the held (not missing) Restart, retiring on delivery rather than queueing, and that the Restart event already exists. All four are applied, and each has a reply. The head after fafcf8e9 only applies that review's wording.

(Written by Claude Opus 5.5)

@kixelated
kixelated enabled auto-merge October 13, 2026 04:13
@kixelated
kixelated added this pull request to the merge queue Oct 13, 2026
Merged via the queue into main with commit 2051754 Oct 13, 2026
3 checks passed
@kixelated
kixelated deleted the plan/interop-main-failures branch October 13, 2026 04:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant