Skip to content

Fix the gate's Windows pruning race, Release's macOS queue, and Windows git-test limits - #233

Merged
kixelated merged 7 commits into
mainfrom
quest/m0/ci-flakes
Oct 28, 2026
Merged

kixelated merged 7 commits into
mainfrom
quest/m0/ci-flakes

Conversation

@kixelated

@kixelated kixelated commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Works on quest/m0/ci-flakes.md. This PR fixes the gate headroom failure and Release's macOS queue, gives the native Windows git tests a file-level time limit, and deletes the finished quest.

Gate headroom: a real Windows bug, not a slow host

The Plan assumed that slow runners admitted only 2 or 3 of the 4 callers. They didn't. The old test's callers discarded any rejection, so I reran the race on the native runners with every outcome recorded. That was 60 rounds per runner on a throwaway diag/ci-flakes branch (run).

  • The ledger always held exactly 4 admissions. Callers crashed 27 times with EFAULT: bad address in system call argument, rm '…\ledger\1'. Every under-admitted round (18 of 60 on windows-11-arm, 2 of 60 on windows-2022) had a caller that published its admission and then crashed. No caller was ever too slow.
  • A focused race confirmed the cause on both runners (run). When two processes rmSync(file, { force: true }) the same file at once, Bun throws EFAULT, where Node throws EPERM. Callers pruning the same superseded ledger version hit this, and remove() only tolerates EPERM/EBUSY.
  • The same race showed that reading a file another process is deleting throws EPERM on Windows in both runtimes. ledger() retried only on ENOENT.

On Windows, quest gate could fail a command whose slot it had just recorded.

Fix (src/cli/gate.ts):

  • remove() also tolerates Bun's EFAULT.
  • ledger() retries a failed read only when a newer version now exists, whatever the error. A file pruned since the listing is always superseded, and a real read error on the newest version still throws.

Test. As decided, "concurrent processes never admit past the headroom, and admit every caller" no longer expects exactly 4 admissions by a deadline:

  • All 12 callers must be admitted.
  • Each admission is checked against the gate's rule: one that finds 4 admissions from the last minute comes SETTLE_MS after the previous one.
  • The ledger must hold only reported admissions, up to the last.
  • The callers share a clock that runs 20 times faster, so spacing the other 8 takes about 2 s.

Validation.

  • On the native runners, 40 runs each (run): the old gate failed 15 of 40 on windows-11-arm and 0 of 40 on windows-2022. The fixed gate failed 0 of 40 on both.
  • Locally, the test catches both a headroom off-by-one and a non-atomic publish, and it passes under 3× CPU oversubscription.

Release's macOS queue

Release's macOS check jobs now run in their own macos job, skipped on pull requests. They share the Linux job's steps through a YAML anchor. Tags, which publish the artifacts, and manual runs still check macOS, and Check's lifecycle (macos-14) still runs on main. publish needs the new job. Check's comment no longer says pull requests rely on Release for macOS. Release hasn't hit this in its last 100 runs; the change applies the decision from kixelated/quest#218 before it does.

Native Windows git-test timeouts: findings

From full-suite and single-file runs on both runners, with every child process timed (run):

  • move.test.ts and landed.test.ts spend about 95% of their time starting git: 8–13 processes per test, at 35–80 ms each, against about 2 ms on Linux.
  • Their slowest tests take 0.8 s on windows-2022 and 1.4 s on windows-11-arm, where Defender real-time scanning is on. Running files in parallel makes no difference: move.test.ts took 10.0–10.4 s alone and 8.6–10.4 s in the full suite on arm.
  • The four timeouts since 2026-10-21 ran about 3× to 20× slower than normal:
  • Three of the four came within 15 minutes during a CI burst on 2026-10-24, on hosts whose other test files were slow too.
  • Ruled out: a longer PATH under bun run, and GIT_CONFIG_NOSYSTEM (run).
  • What does help: maintenance.auto=false saves 15–25 ms per commit, and copying a prebuilt repository (about 35 ms) beats init + add + commit (140–185 ms).
  • The start.test.ts timeouts in the Plan came from kixelated/quest#167's branch on 2026-10-21. That PR merged with copied fixture repositories and a 30 s file timeout, and start hasn't timed out since.

Native Windows git-test timeouts: time limit

Fewer git processes can't absorb a 20× host stall within vitest's 5 s. The user chose (2026-10-26) a file-level limit over cutting git processes: move.test.ts and landed.test.ts now allow 30 s, as start.test.ts does. That is about 20× their slowest test on windows-11-arm, and still catches a hang. Cutting git processes (copied fixture repositories, maintenance.auto=false, batched lookups) would still make quest mv and quest start faster on Windows, and is listed below as a follow-up.

Choices made without asking

  • Fixed the gate. The Plan rejected treating under-admission as a gate bug, on the premise that slow hosts caused it. The runs showed a crash instead, and the new test requires every caller to be admitted, so it would fail on Windows without the fix.
  • Faster clock in the race. The headroom race uses a clock 20 times faster rather than fewer callers. That keeps 12 concurrent racers and a 2 s test.
  • YAML anchor in Release. The macOS checks moved to a job sharing steps through an anchor, rather than a dynamic matrix with a runner lookup.

Validation

  • just check and just test pass locally, and so does quest check --base origin/main.
  • On the native runners, this branch's Windows Check steps (check, test, build, lifecycle) ran 3 times per runner, and all 6 passed (run).

Suggested follow-ups

  • Report Bun's EFAULT from concurrent rmSync(file, { force: true }) on Windows upstream (needs approval to post outside kixelated/moq-dev). Node reports EPERM.
  • Cut the CLI's and tests' git processes on Windows: batch the CLI's repeated git lookups, and use copied fixture repositories with maintenance.auto=false in tests. quest mv runs git remote get-url origin twice plus git config --get remote.origin.url, and quest start runs similar sets.

The throwaway diag/ci-flakes branch is deleted.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 4 commits October 26, 2026 13:41
Quest-Claim: 9cf7cf4076519f9c
The race asserted exactly 4 admissions before a deadline, which slow
native Windows runners missed. It now admits all 12 callers on a clock
running 20 times faster, and checks each admission against the rule.

ci(release): skip macOS checks on pull requests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Callers that prune the same superseded version at once made Bun's rmSync
throw EFAULT on Windows, crashing a caller after its admission was
recorded. A ledger read of a version being pruned throws EPERM there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Fail CI checks only on real breakage Fix the gate's Windows pruning race and Release's macOS queue Oct 26, 2026
Each test spawns git up to a dozen times, and a busy Windows runner has
stretched them 3-20x past their usual 0.8-1.4 s. Like start.test.ts, they
now allow 30 s, which still catches a hang.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Fix the gate's Windows pruning race and Release's macOS queue Fix the gate's Windows pruning race, Release's macOS queue, and Windows git-test limits Oct 27, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Decided (user, 2026-10-26): give test/move.test.ts and test/landed.test.ts a file-level time limit like test/start.test.ts (30 s), rather than cutting git processes to fit 5 s. Done in this PR, and the quest is deleted.

(Written by Claude Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review October 27, 2026 00:03
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-28T18:45:29.441684Z 79ca67c New commits
🔒 Security Review ✅ Completed 2026-10-28T18:46:56.187829Z 79ca67c New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: e48273facfbab009ce1c22ea452acff85ffbe16f.

No actionable bugs found in the eight-file diff and surrounding gate, test, and workflow logic. Overall direction is sound: the small ledger helper preserves atomic publication, shared workflow steps avoid duplication, and the scoped timeouts follow the recorded decision without unrelated refactoring. No simpler structural change appears necessary.

Verification limits: GitHub-only static review; I did not run tests or reproduce the Windows races. Check and Release were still running when inspected, so this is not a test-pass or merge-readiness claim. The independent Codex summary also reports no findings.

(written by OpenAI)

Resolve quest/m0/README.md by taking main's Required list without the
finished CI flakes quest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merged main into this branch (cc47b73). The one conflict, in quest/m0/README.md's Required list, was resolved by taking main's list without the finished CI flakes line. The PR's diff against main is byte-identical to the reviewed head e48273f. just check and just test pass locally, and required CI passes on the new head.

Because the conflict was resolved by hand, the new head needs a fresh review before landing. The automated reviewer has not re-reviewed after the push, so this is waiting on a maintainer decision.

Today's native Windows failures on other PRs (gate.test.ts "concurrent processes never admit past the headroom", expected 4 admissions but got 3) are the ledger pruning race this PR fixes.

(written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge summary: head cc47b73 merges main into the reviewed e48273f. The diff against main is byte-identical to the reviewed one, and the only conflict was in the quest README link list. Codex gave a 👍 after the push, required CI is green (12/12), and quest check --base origin/main --branch quest/m0/ci-flakes passes. The maintainer chose to land it in /quest-complete. It fixes the gate ledger pruning race behind today's optional Windows gate.test.ts failures.

(written by Claude Opus 5.5)

@kixelated
kixelated enabled auto-merge (squash) October 28, 2026 18:45
@kixelated
kixelated merged commit 5969018 into main Oct 28, 2026
12 checks passed
@kixelated
kixelated deleted the quest/m0/ci-flakes branch October 28, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant