Skip to content

Stop the lifecycle CI jobs failing at random - #218

Merged
kixelated merged 4 commits into
mainfrom
quest/m0/lifecycle-flakes
Oct 24, 2026
Merged

kixelated merged 4 commits into
mainfrom
quest/m0/lifecycle-flakes

Conversation

@kixelated

@kixelated kixelated commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Completes quest/m0/lifecycle-flakes.md: the lifecycle CI jobs no longer fail for reasons outside the code under test.

Windows EBUSY

Cause. On windows-11-arm hosted runners, GitHub's hosted-compute-agent starts an unsigned helper,
%TEMP%\provjobd.exe<N>. About every 15–35 s it opens the image of a running process. It first opens it
without delete sharing for 300–450 ms (RW-), then with delete sharing for about 3 s (RWD). If that scan
caught the check's last quest.exe run, rmSync(work) hit the RW- window while removing the check's copy at
work\tool bin\quest.exe, and failed with EBUSY. windows-2022 has no failures, and its Defender real-time
protection is off. The JS build in Check never failed because bun.exe lives outside the temp directory.

Reproduction. I used a throwaway diag/lifecycle-ebusy branch, now deleted. It looped the check on native
runners, running Sysinternals Handle when the check failed and polling it in the background:

  • Before: 8 EBUSY in 390 runs on windows-11-arm (0 in 80 on windows-2022). Each time, the only thing left
    was tool bin\quest.exe, held by provjobd.exe. The failing rm started 0 ms after the last quest uninstall
    exited, inside a polled RW- window (06:22:23.003–23.461, rm at 23.067). It was removable a moment later.
  • After: 0 failures in 300 runs of lifecycle-check.mjs and 120 runs of install-check.ps1 -Method archive on
    windows-11-arm.

Fix. Nothing deletes an executable the check just ran. There are no retries, sleeps, or ignored errors.

  • lifecycle-check.mjs runs the caller's binary in place instead of copying it into the directory it deletes.
  • lifecycle-check.sh copies the binary into "$work/tool bin/" first, which keeps the Linux/macOS coverage of a
    binary path with spaces.
  • install-check.ps1 puts its scratch under RUNNER_TEMP, which the runner empties after every job, and no
    longer removes it itself. It already installs to a path with spaces (quest install <guid>). The script is
    CI-only (Release and Install check), so it now refuses to run without RUNNER_TEMP.

macOS queue timeout

When hosted macOS runners are short, the lifecycle (macos-14) job waits in the queue until timeout-minutes
cancels it. That happened on kixelated/quest#191 (queued 15:40:13,
cancelled 15:55:14, no runner). The job then shows as failed. Pull requests now run lifecycle on
ubuntu-latest only. macos-14 still runs on pushes to main and on manual runs.

macOS stays covered on pull requests by Release. Its native check (aarch64-apple-darwin) and
check (x86_64-apple-darwin) jobs run the lifecycle check on any PR that touches src/, scripts/, or the
lockfiles. The only macOS check missing from PRs is the flake's own Nix build on Darwin, which now runs on main.

Choices made without asking

  • Of the three macOS options, I picked "drop it from pull request runs". A shorter queue limit still shows a
    failure. A cancelled job can't be made to report as skipped.
  • install-check.ps1 now needs RUNNER_TEMP, so running it locally is unsupported. It never had any local use.
  • Check's native Windows job still runs dist/quest.js from the checkout. That path has no spaces, so the
    spaces coverage on Windows now comes from the native binary that Release and Install check run.

Validation

  • just check and just test passed locally (both lifecycle builds included), as did quest check --base origin/main.
  • Release passed on this branch, including native windows-11-arm and macOS lifecycle checks.
  • The diagnostic loops are described above.

Suggested follow-ups

  • Release's macOS check jobs can hit the same queue timeout. On a PR, that shows as a failed (non-required)
    Release check.
  • Native Windows unit-test flakes, unrelated to the lifecycle, have caused most recent Check failures:
    test/gate.test.ts "concurrent processes never admit past the headroom" (5 of the last 15 failed Check runs),
    test/move.test.ts "refuses a claimed quest" timing out at 5 s, and test/start.test.ts timeouts.
  • quest start failed once with "git push to origin failed" even though the remote printed the "Create a pull
    request" hint. A retry worked. The error dropped the porcelain stdout, which would have explained it.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 4 commits October 23, 2026 23:04
Hosted macOS runners can queue past the job's timeout, which reports a job
that never ran as failed. Pull requests keep native macOS coverage through
Release's lifecycle checks; main still checks the flake's macOS build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On windows-11-arm, GitHub's hosted runner agent (provjobd, started by
hosted-compute-agent) periodically opens the images of running processes
without delete sharing for a few hundred milliseconds. When that caught the
check's last quest.exe run, removing the check's copy of the binary failed
with EBUSY. Run the caller's binary in place, and let the runner discard
install-check.ps1's scratch under RUNNER_TEMP rather than deleting it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review October 24, 2026 07:21
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-24T07:23:23.686645Z 7c8f9c2 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-24T07:23:57.021077Z 7c8f9c2 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7c8f9c206f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/check.yml Resolved
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging as a squash at 7c8f9c2.

  • Windows EBUSY: the root cause is fixed. Nothing deletes an executable the check just ran, and there are no retries or sleeps.
  • macOS: the lifecycle job runs on pushes to main and on manual runs. On PRs, Release covers macOS for changes under src/, scripts/, and the lockfiles.
  • Maintainer decision on the Codex thread: keep this as is. PRs that touch only Nix files get no pre-merge macOS check. Thread resolved.
  • CI passed (16 checks). quest check --base origin/main passed on the head.

(Written by Claude Opus 5.5)

@kixelated
kixelated merged commit 88b2038 into main Oct 24, 2026
16 checks passed
@kixelated
kixelated deleted the quest/m0/lifecycle-flakes branch October 24, 2026 14:44
kixelated added a commit that referenced this pull request Oct 24, 2026
Resolve the quest/m0/README.md conflict by dropping both finished quests (lifecycle-flakes merged in #218, worker-tests finishes here).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant