[kixelated]

Finishes and deletes quest/m1/test-flakes-2/redundant-failover-drift.md.

Summary

export::redundant_importers_switch_through_bufferbloat_on_lite_07 had two independent causes behind the same "stitching export exited" message. Both were found from the exporter's stderr and per-frame jitter timing on failing runs.

1. The jitter clock's "drift" was the container consumer metering a backlog. After the test kills the importer the export just switched to, the relay keeps the dead route until the QUIC idle timeout, so the newest content stops advancing. The dead source's audio backlog has every other group missing (about 120 ms holes). Each hole's monotonic budget (max_delay = delay/2 = 1.5 s) started when the read cursor reached it, so the backlog drained one group per 1.5 s: the log shows media skipped upstream ... skipped=120ms exactly 1.502 s apart. jitter::Buffer read that paced trickle as a source running at 0.17x and failed (-2148785 ppm, -6508448 ppm). The jitter behaves to its documented contract (moq-dev/moq#5490, moq-dev/moq#5714): a backlog read at once gives one floor. The metering upstream of it was the bug.
   • Fix: a hole is now late from when the first group past it arrived, not from when the cursor reached it. Holes behind a stalled edge expire together. Each pending group carries the earliest arrival among it and the groups past it, copied from the next group on insert (arrivals only move forward), so a hole reads it from the head in O(1) instead of scanning a peer-controlled backlog per hole. Applied to both rs/moq-mux/src/container/consumer.rs and js/hang/src/container/consumer.ts, which moq-dev/moq#5999 introduced together.
2. json: old. When a relay resets a catalog group, the reset arrives as Error::Stream(StreamError::Old). moq_json::snapshot::Consumer only treated the local Error::Old/Evicted/Lagged/GroupTooLarge as a lost group, so the export failed. moq-dev/moq#6204 fixes the same match in the moq-json snapshot and window consumers and moq-flate's snapshot consumer, classifying by StreamError::from, so this PR no longer carries its own copy of that fix.
3. Per the quest plan, landed after the cause fixes: redundant importers in the test run with --quic-idle-timeout 2s --quic-keep-alive 500ms. The failover test also prints the exporter's stderr when it exits early.
4. Quest bookkeeping: main moved the quest to quest/m0/redundant-failover-drift.md; that copy and its quest/m0/README.md entry are deleted, and the references in quest/m1/test-flakes-2/README.md and quest/m1/audit-compile-time.md now point at this PR (quest/m1/audit-test-cost.md finished on main).

None of the quest's three options was the cause. Option 1 (jitter) would have undone #5490's fail-fast for sources far off real time; option 2 (follower stale-media switching) belongs to route health.

Public API / wire impact

• Public API: none. Behavior: a container consumer now times a sequence hole from the first later group's arrival. (The snapshot/window consumer change now lands in moq-dev/moq#6204.)
• Wire: none.

Tests

Each regression test failed on the original code (checked by restoring the old source) and passes with the fix:

• moq-mux container::consumer::tests::holes_behind_a_stalled_source_share_one_budget: on the old code it read ["gap", "frame 2"] after one budget instead of all three groups.
• JS consumer.holes.test.ts: "the holes a stalled source left behind share one budget".
• a_hole_is_late_from_the_earliest_arrival_past_it (Rust) and "a hole is late from the earliest arrival past it" (JS): a group arriving below an earlier one inherits that arrival. Each fails without the copy on insert.

Checks run:

• Before the fix, the bufferbloat test failed 2 of 13 local runs (one ppm, one json: old). With both fixes and the default idle timeout, it passed 12 of 12.
• With the 2 s idle timeout: the three redundant-importer tests passed 7 of 7 runs (passthrough 4.2 s, survivor-and-back 15.5 s, bufferbloat 29.6 to 33.2 s).
• just check: passed (4,286 + 516 + 216 + 123 + 10 Rust tests, JS tests, lint). It caught rustfmt and Biome formatting, fixed in this push.
• quest check: passed.
• At f14d324 (O(1) hole lateness): cargo nextest -p moq-mux -p moq-cli (1778 passed), moq-mux clippy, js/hang tests (278 passed), tsc and Biome pass.
• After merging main again at 6b5fd8e (quest conflicts only): moq-mux nextest (1586 passed), js/hang tests and tsc, and quest check pass.
• After merging main at 1b5621e: cargo nextest -p moq-mux -p moq-cli (1777 passed), the three ignored redundant-importer tests 3 of 3 under --stress-count 3 (bufferbloat 29.9 to 35.2 s), js/hang tests (277 passed), and quest check all pass. nix flake check cannot run on this host (the github-runner image build needs user namespaces the sandbox refuses); the diff touches no Nix.

Bufferbloat test duration

It still runs about 30 s. moq-dev/moq#6192 has since moved the real-time export tests behind #[ignore] into the nightly just rs cli-export leg, so this cost no longer lands on just check. Timed per phase on one run: first output 4.9 s, first switch 15.6 s, second 21.2 s, third 33.1 s, with 3 to 8 s viewer output gaps at each switch. The downlink is shaped to 160 kbit/s against a roughly 400 kbit/s source, so every switch waits for the new importer's keyframe group to cross a link below the media rate. That cost is the scenario itself, and the test runs real subprocesses on wall time, so a paused clock does not apply. Shortening it means a faster link, which weakens the bufferbloat coverage, so it stays.

Decisions (confirmed by the maintainer)

1. Hole budget anchor (behavior change to fix(mux): bound sequence hole waits by max delay #5999's rule): a hole is late from the earliest arrival among the groups past it.
   • ✅ As implemented. It is the moment the hole became observable, and it matches "a stream that lost the delivery race".
   • Keep per-cursor timing and fix only the TS jitter. Rejected: the jitter cannot tell a metered backlog from a slow source.
   • Trade-off: a hole behind a long-blocking current group may now expire as soon as the cursor reaches it, rather than getting another full budget.
2. Wire-form matching:
   • Superseded: moq-dev/moq#6204 classifies by StreamError::from, so this PR dropped its inline arms.
3. Bufferbloat test duration:
   • ✅ Keep the 160 kbit/s scenario and its ~30 s cost (see above).
   • Raise the link rate to finish faster, at the cost of the below-media-rate coverage.

Suggested follow-ups

• A slow link-rate drain (not metered by holes) early in a generation is still the jitter's documented known limit from fix(moq-mux): a source the output cannot follow fails the export promptly #5490. Cross-epoch failover without waiting for the idle timeout belongs to route health.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 8 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 61e53dcf-5c96-4ff3-aa86-1e447b9fc37f

📥 Commits

Reviewing files that changed from the base of the PR and between 2d73eea and 1cac2a0.

📒 Files selected for processing: 8

• js/hang/src/container/consumer.holes.test.ts
• js/hang/src/container/consumer.ts
• quest/m0/README.md
• quest/m0/redundant-failover-drift.md
• quest/m1/audit-compile-time.md
• quest/m1/test-flakes-2/README.md
• rs/moq-cli/tests/integration/export.rs
• rs/moq-mux/src/container/consumer.rs

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-22T09:54:30.819815Z	1cac2a0	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[kixelated]

Merged origin/main again (a587c85). The only conflict was quest/m1/test-flakes-2/README.md: main finished the draft-16 cancel race and this PR finishes redundant failover drift, so both children leave Required and Relay bench holder stays. No code changed since the Codex-reviewed 8226249 beyond main merges.

Checks: quest check ok, just check passed, and the redundant importer tests passed 6 of 6 under nextest --stress-count 6 (plus the just check run).

(Written by Claude Opus 5.5)

[kixelated]

Merged origin/main (245a378).

• Conflicts: quest/m1/test-flakes-2/README.md (kept main's list minus this quest) and quest/m1/resume-outage.md (deleted on main; took the deletion). A new reference to the deleted quest in quest/m1/audit-compile-time.md now points at this PR.
• Dropped the moq-json/moq-flate lost-group changes: moq-dev/moq#6204, now in the merge queue, fixes the same match by classifying through StreamError::from. This PR now carries only the container hole-budget fix (Rust and JS), the failover test's idle-timeout and stderr changes, and quest bookkeeping. The description is updated.
• Checks: Rust test-changed (3627 + 516 + 224 passed, including both redundant-importer tests), JS test-changed, lint, quest check, markdown and taplo all pass. nix flake check cannot run on this host (the github-runner image build needs user namespaces the sandbox refuses); the diff touches no Nix.

(Written by Claude Opus 5.5)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 245a378edc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1b5621eef8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[kixelated]

Merge summary for 6b5fd8e:

• Fixes the redundant TS failover stall. A killed importer behind bufferbloat no longer stalls the consumer: hole lateness is measured from the earliest pending arrival, and each pending group carries the earliest arrival among it and later groups. A hole reads it in O(1) (f14d324), mirrored in js/hang.
• Codex threads are resolved. The quadratic rescan is fixed in f14d324. The json: old lost-group errors are fixed on main by moq-dev/moq#6204, so they stay out of this PR.
• Codex reviewed this head with no findings. Since f14d324, only an origin/main merge and quest edits landed: the finished quest/m0/redundant-failover-drift.md is removed, along with its references.
• Required checks run in the merge queue.

(Written by Claude Opus 5.5)

[kixelated]

Re-queued at the same head (6b5fd8e). The 05:39Z removal was manual with no recorded reason: no merge-group run had started for this PR, no new review finding or failing check followed, and no open PR supersedes it. Main's six newer commits merge cleanly (test merge passes quest check) and touch nothing in the container consumers.

(Written by Claude Opus 5.5)