fix(ci): repair red nightly gates and the ACME port race by kixelated · Pull Request #6056 · moq-dev/moq

kixelated · 2026-10-20T06:17:36Z

Problem

The checkpoint in #5976 left one queue failure unexplained (ACME's rate-limit test hit its 30 s watchdog) and only looked at the Nightly's stalled benchmarks. The latest Nightly had 28 failed jobs from eight independent causes, some red for days, so new regressions went unnoticed.

Approach

One commit per fix:

ACME port race (rs/moq-relay/src/acme/tests.rs): two tests bound a port, dropped it, and assumed it stayed dead. Another test process can bind the released port; a listener there that never answers stalls the challenge dial for the whole 30 s budget. Rebinding the port locally reproduced the exact 30 s stall. The tests now hold a bound, non-listening socket, which refuses every dial and keeps the port reserved. The helper asserts no other listener can take it.
send-depth and all six fuzz targets (rs/moq-net/src/stats.rs): broken since #5773 on 2026-10-17 (bisected with nightly 2026-10-19). That PR made a stats session and its wire counters reference each other, so every IETF subscriber future repeated a recursive auto-trait proof past the default recursion limit. Each shared session context is now a SendCell, as #5682 did for the origin graph: the proof runs once at construction. No allocation or API change.
Generator comparison (bench/ffi-gen/Cargo.lock): broken since #5911 moved the Nix wasm-bindgen CLI to 0.2.129. ffi-gen keeps its own lock, which was still at 0.2.127. It now matches the workspace's wasm crates.
moq-uring host clock check (rs/moq-uring/src/clock.rs): host_tracks_moq_clock, new in #5926, allowed a fixed 1 ms calibration error. A loaded just check here failed it with the clock 1.5 ms ahead, because a preemption between the two calibration reads stretches the error. The test now measures the calibration's window and uses that as its slack.
Quests: the run is complete, so audit-test-cost moves to the end of m1's Required with a rewritten Plan. Three new quests cover what is not fixed here (below).

The other Nightly causes were already fixed on main or have owners: the CI selection count (#5976), the stats bench import, Windows moq-video (#5966), release-build linking (release-link), and Rust advisories (audit-deps). The scheduled Interop failure (stale wire-compat pin) was re-pinned on main.

Impact

Public API: none. Wire: none. stats::Session's private field changes type only.
Test-only and lockfile changes otherwise.

Validation

ACME: all 8 tests pass. The rate-limit test takes 6.2 s, set by rustls-acme's own backoff.
sh/rs/send-depth.sh's check (-D recursion_depth_exceeding_limit on nightly 2026-10-19) fails on main and passes here, including after merging current main. moq-net stats unit tests pass on stable.
ffi-gen: the UniFFI wasm build plus the dev shell's wasm-bindgen succeed.
just check: Rust tests pass (7,657 in one run). Two of three later loaded runs failed only on the pre-existing redundant_importers_switch_through_bufferbloat_on_lite_07 flake (planned below); the modules after it (toml, nix, justfile, gh, py with 102 passed) were rerun separately and pass. quest check: 429 documents ok.
Not run: the full Nightly and macOS/Windows Platform jobs.

Decisions

The maintainer was away and authorized the recommended course, so each of these was chosen provisionally under it. Any can be revisited before merge.

Complete this run now. Every finding is fixed or planned, so the successor goes to m1 rather than another m0 checkpoint.
New quests go in m1, not m0. m0's goal is relay hardening and identity.
SendCell rather than boxing for the stats graph. It matches #5682 and costs no allocation.
Hold the redundant-importer speedup. Giving the importers a 2 s QUIC idle timeout cut them from 15/29/37 s to 4/16/30 s and stopped the bufferbloat variant flaking, but it only shortens a stall that seems to expose a TS export bug, so it is recorded in the new quest instead.
Redundant failover drift goes first in test-flakes-2. It likely hides a TS export bug and flakes more often than the others.

Follow-ups

Redundant failover drift: redundant_importers_switch_through_bufferbloat_on_lite_07 failed 3 of 10 isolated local runs on main and 2 of 3 loaded just check runs here. When the importer is killed, the jitter clock reads the stalled route as -6e6 ppm drift and fails the export instead of switching to the survivor. Likely a product bug. Recommend starting this first.
Nightly media players: neither row has passed since #5640. Playwright WebKit lacks the decoders (recommend dropping that row), and Firefox stalls at the audio gesture gate (recommend reproducing on the arm runner).
macOS QUIC connects: close_code and fixed_target time out on macOS only. 120 stressed Linux runs passed.
#6031 makes the Nix CLI follow the root Cargo.lock. bench/ffi-gen's separate lock can still drift, so consider covering it there.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The rate-limit and unreachable-directory tests bound a port, dropped it, and assumed nothing listened there afterwards. Under a parallel test run another process can bind the released port; a listener there that never answers stalls the challenge dial until the 30 s budget, which is how queue run 39216656008 failed. Rebinding the port reproduced that 30 s stall locally. A bound socket that never listens refuses every dial and keeps the port reserved, which each test now asserts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Nightly's send-depth gate and every fuzz target have failed since #5773 gave a stats session and its wire counters references to each other: each IETF subscriber future repeated that recursive auto-trait proof and overflowed the default recursion limit. Bisected to that commit with nightly 2026-10-19. Wrap each shared session context in SendCell, as #5682 did for the origin graph, so the proof runs once at construction. No allocation or API change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The nightly generator comparison has failed since #5911 moved the workspace and the Nix wasm-bindgen CLI to 0.2.129: bench/ffi-gen keeps its own lock, still at 0.2.127, whose schema the CLI refuses. Update it and the wasm crates that pin it exactly to the workspace's versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t audit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

`host_tracks_moq_clock` allowed a fixed 1 ms between the calibrated clock and moq-clock, but the calibration's error is half the gap between its two reads, which a preemption stretches. A loaded `just check` failed it with the host clock 1.5 ms ahead. Measure the calibration's window and use that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

coderabbitai · 2026-10-20T06:19:09Z

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 11 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 6e88aebd-15b6-4b90-ad1d-a93dd59f4d7e

📥 Commits

Reviewing files that changed from the base of the PR and between 877d6fe and 733f3a0.

⛔ Files ignored due to path filters: 1

bench/ffi-gen/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing: 13

quest/m0/README.md
quest/m0/audit-perf.md
quest/m0/audit-test-cost.md
quest/m0/ci/audit.md
quest/m1/README.md
quest/m1/audit-test-cost.md
quest/m1/media-engines-nightly.md
quest/m1/test-flakes-2/README.md
quest/m1/test-flakes-2/macos-quic-connect.md
quest/m1/test-flakes-2/redundant-failover-drift.md
rs/moq-net/src/stats.rs
rs/moq-relay/src/acme/tests.rs
rs/moq-uring/src/clock.rs

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

chatgpt-codex-connector · 2026-10-20T06:19:14Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-20T06:47:04.018458Z	`733f3a0`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

kixelated · 2026-10-20T06:25:47Z

Merging under the recommended course the maintainer authorized while away.

Summary: fixes the ACME port race (tests hold a bound, non-listening socket), the send-depth and fuzz recursion-limit break from #5773 (SendCell for shared stats session context), the stale bench/ffi-gen wasm-bindgen lock, and the moq-uring clock test's fixed 1 ms slack. Completes the m0 test-cost audit and plans the rest in m1.

Review: Codex reviewed head c9b84a2 with no findings and a thumbs-up. CodeRabbit was rate-limited (optional).

Decisions: the five provisional decisions in the PR body stand as recorded (complete this run; new quests in m1; SendCell over boxing; hold the redundant-importer speedup; redundant failover drift first in test-flakes-2).

CI: Check and Test run only in the merge queue. The known redundant_importers_switch_through_bufferbloat_on_lite_07 flake may hit the queue; it is tracked in quest/m1/test-flakes-2/redundant-failover-drift.md.

(Written by Claude Opus 5.5)

# Conflicts: # quest/m0/README.md # quest/m1/README.md

kixelated · 2026-10-20T06:43:59Z

The queue dropped this PR because #6050, ahead of it, edited the same quest/m0/README.md and quest/m1/README.md lines. Merged origin/main (733f3a0): m0 drops both audit entries, and m1 lists the coverage audit and then the test-cost audit at the end. No code files changed on main since this PR's base; quest check passes. Re-enqueuing.

(Written by Claude Opus 5.5)

kixelated and others added 12 commits October 19, 2026 21:39

quest(m0): claim audit-test-cost

904f0c4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

quest(m1): plan the macOS QUIC flake and the nightly media players

c10ec9c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into audit-test-cost-2

a9e191e

quest(m1): plan the redundant failover drift and advance the test cos…

33ef77d

…t audit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

quest(m1): record the uring calibration fix in the test cost audit

bec3685

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

quest(m1): rank the redundant failover drift first among the flakes

2e68149

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into audit-test-cost-2

f365180

quest(m0): point the perf audit at the merged JS stall fix

c9b84a2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kixelated marked this pull request as ready for review October 20, 2026 06:19

kixelated added this pull request to the merge queue Oct 20, 2026

kixelated mentioned this pull request Oct 20, 2026

feat(demo): add a CARLA teleop demo #5621

Merged

github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 20, 2026

Merge remote-tracking branch 'origin/main' into audit-test-cost-merge

733f3a0

# Conflicts: # quest/m0/README.md # quest/m1/README.md

kixelated enabled auto-merge October 20, 2026 06:44

kixelated added this pull request to the merge queue Oct 20, 2026

Merged via the queue into main with commit dcd7d2c Oct 20, 2026
4 checks passed

kixelated deleted the quest/m0/audit-test-cost branch October 20, 2026 07:00

This was referenced Oct 20, 2026

quest: plan load flakes, GOAWAY follow-ups, and the moq-quic transport direction #6083

Merged

quest: audit the tree against main (2026-10-22) #6273

Merged

Navigation Menu

Repository navigation

fix(ci): repair red nightly gates and the ACME port race - #6056

fix(ci): repair red nightly gates and the ACME port race#6056
kixelated merged 13 commits into
mainfrom
quest/m0/audit-test-cost

kixelated commented Oct 20, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

Review limit reached

Uh oh!

chatgpt-codex-connector Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

kixelated commented Oct 20, 2026

Uh oh!

kixelated commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

kixelated commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Approach

Impact

Validation

Decisions

Follow-ups

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

chatgpt-codex-connector Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

kixelated commented Oct 20, 2026

Uh oh!

kixelated commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

kixelated commented Oct 20, 2026 •

edited

Loading

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 20, 2026 •

edited

Loading