Repository navigation
quest: plan making tests fit their timeout on the queue runner - #497
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d background work Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 18 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
Reviewed the whole diff (2 files, 2 commits) plus the repo's format and evidence contract: this is a plan-only PR, so correctness is the accuracy of the quest document and its grounding.
- New quest
[M] Worker route tests fit their timeout(quest/m0/queue-watcher/worker-test-timeouts.md) with the required## Goaland a## Planthat records the settled decisions, the failure evidence, and implementation leads for the implementer. - Epic index row (
quest/m0/queue-watcher/README.md) inserted in the## Requiredlist between "Harden the queue runners" and "Daemons prune their logs", matching the house- [link](...) - outcomestyle.
Every checkable claim verifies against the tree and GitHub:
- Format:
bun run --silent quest checkpasses (79 documents ok); headings, size tag, and the root-absolute /quest-m0/queue-watcher links resolve. - Code facts:
waitUntilbackground work incloudflare/src/github/webhook.ts;isolate: falseand the/dev/shmTMPDIR line incloudflare/vitest.config.ts; the single-workerd pool comment from #112;scripts/gate-bench.sh; thejust worker-test-isolatedrecipe;github.test.tsreally does stub no globalfetch(its only fetch use is a per-service mock); thetest/review.test.tscase name matches verbatim. - Run facts:
gh run list --event merge_group --status failure --created 2026-11-04shows exactly 10 failed queue runs that day, and all cited run IDs (41901538568, 41905766925, 41886648526, 41879887537) are among them. The "all 22checkjobs ran the whole Worker suite" lead is sound:check.yml'scheckjob runsjust checkand thenjust test, whose_testrecipe ends injust worker-test, onubuntu-latest. - Process facts: the three-flaky stop rule is
REQUEUE_LIMIT = 3insrc/cli/land-fixer.ts:228, so "#481 hit its three-flaky stop rule" is accurate; the runner-ownership correction (kixelated/nixos#19 added it,runners.mdonly restricts access) matches the cutover records.
The plan's guidance (measure host load before blaming the tests; hunt the webhook-started waitUntil work that outlives its test) follows directly from that evidence, and the body's recommendation to drop the root-suite timeout follow-up unless it recurs matches the guide's "drop one you would call optional".
glm-5.3-flash ｜ 𝕏
|
quest land watch: the merge queue removed this pull request at f038009 because its The land fixer works on it in the background, and the watcher re-enqueues it at the back of the queue once it is fixed or judged flaky, or hands it back here with what it needs.
…pect(empty.queue).toEqual([]);
�[90m �[2m❯�[22m processTicksAndRejections unknown:�[2m7:39�[22m�[39m
�[31m�[2m⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/2]⎯�[22m�[39m
�[41m�[1m FAIL �[22m�[49m test/review.test.ts�[2m > �[22ma reviewer's usage limit exits 3 with its reset, and records no round
�[31m�[1mError�[22m: Test timed out in 5000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".�[39m
�[36m �[2m❯�[22m <anonymous> test/review.test.ts:�[2m379:6�[22m�[39m
�[90m377|�[39m const round = review({ root: s.repo, signal: controller.signal }, { e…
�[90m378|�[39m while (!existsSync(pid)) await new Promise((resolve) => setTimeout(re…
�[90m379|�[39m controller.abort();
�[90m |�[39m �[31m^�[39m
�[90m380|�[39m await expect(round).rejects.toThrow(/abort/i);
�[90m381|�[39m // The round fails only once the CLI exited, so nothing outlives it i…
�[90m �[2m❯�[22m processTicksAndRejections unknown:�[2m7:39�[22m�[39m
�[31m�[2m⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/2]⎯�[22m�[39m
�[2m Test Files �[22m �[1m�[31m2 failed�[39m�[22m�[2m | �[22m�[1m�[32m27 passed�[39m�[22m�[90m (29)�[39m
�[2m Tests �[22m �[1m�[31m2 failed�[39m�[22m�[2m | �[22m�[1m�[32m1073 passed�[39m�[22m�[2m | �[22m�[33m5 skipped�[39m�[90m (1080)�[39m
�[2m Start at �[22m 14:21:56
�[2m Duration �[22m 1589.77s�[2m (tests 92%, transform 6%, import 1%)�[22m
##[error]Error: Test timed out in 30000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
❯ <anonymous> test/land.test.ts:1591:32
❯ processTicksAndRejections unknown:7:39
##[error]Error: Test timed out in 5000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
❯ <anonymous> test/review.test.ts:379:6
❯ processTicksAndRejections unknown:7:39
error: script "test" exited with code 1
error: recipe `_test` failed on line 33 with exit code 1
error: recipe `test` failed on line 30 with exit code 1
##[error]Process completed with exit code 1.
Post job cleanup.
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git version
git version 2.55.0
Temporarily overriding HOME='/var/cache/github-runner-work/_temp/24332dd9-ebc6-41a3-8e6e-d9ba95256545' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --global --add safe.directory /var/cache/github-runner-work/quest/quest
Removing SSH command configuration
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp core\.sshCommand
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
Removing HTTP extra header
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
Removing includeIf entries pointing to credentials config files
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
Post job cleanup.
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git version
git version 2.55.0
Temporarily overriding HOME='/var/cache/github-runner-work/_temp/5afb61f4-2030-47d3-98e6-9a69410f57ef' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --global --add safe.directory /var/cache/github-runner-work/quest/quest
Removing SSH command configuration
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp core\.sshCommand
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
Removing HTTP extra header
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
Removing includeIf entries pointing to credentials config files
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
A job completed hook has been configured by the self-hosted runner administrator
##[group]Run '/nix/store/gx2v6q84b0w43bafm0whkdx93qyxbpps-ci-cache-gc.sh/bin/ci-cache-gc.sh'
shell: /nix/store/1mv3qz005gkbalxfghk4y3s8ayv4d2dl-bash-interactive-5.3p15/bin/bash --noprofile --norc -e -o pipefail {0}
##[endgroup]
evicted 0 objects and 0 action results (0 B logical); 50.0 GiB logical remain
dropped 1 stale checkout records
Cleaning up orphan processes
Current runner version: '2.338.2'
##[group]Runner Image Provisioner
Hosted Compute Agent
Version: 20261021.618
Commit: 21505ad2541d5be6135ed84fc33697146cf75b8e
Build Date: 2026-10-21T00:45:21Z
Worker ID: {6788a9f0-468b-4aef-84f3-d881e0908de9}
Region: northcentralus
Cloud: Azure
##[endgroup]
##[group]Operating System
Ubuntu
24.04.5
LTS
##[endgroup]
##[group]Runner Image
Image: ubuntu-24.04
Version: 20261025.353.1
Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20261025.353/images/ubuntu/Ubuntu2404-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20261025.353
##[endgroup]
##[group]GITHUB_TOKEN Permissions
Contents: read
Metadata: read
PullRequests: read
##[endgroup]
Secret source: Actions
Cache mode: write
Prepare workflow directory
Prepare all required actions
Complete job name: Queue
##[group]Run [[ "$RESULT" == success ]] || { echo "::error::the queue build ended $RESULT" >&2; exit 1; }
�[36;1m[[ "$RESULT" == success ]] || { echo "::error::the queue build ended $RESULT" >&2; exit 1; }�[0m
shell: /usr/bin/bash -e {0}
env:
RESULT: failure
##[endgroup]
##[error]the queue build ended failure
##[error]Process completed with exit code 1.
Cleaning up orphan processes
(written by quest land watch) |
|
codex / Codex land fixer note:
(written by codex / Codex) |
|
codex / Codex land fixer: �[41m�[1m FAIL �[22m�[49m test/land.test.ts�[2m > �[22mhost daemon�[2m > �[22mopens the halt issue when the base fixer's launch was lost
�[31m�[1mError�[22m: Test timed out in 30000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".�[39m
�[36m �[2m❯�[22m <anonymous> test/land.test.ts:�[2m1591:32�[22m�[39m
�[90m1589|�[39m forge.repo.pulls[0].landing = false;
�[90m1590|�[39m forge.advance({ "fixed.txt": "yes" });
�[90m1591|�[39m writeFileSync(options.sandbox, "#!/bin/sh\nexit 0\n", { mode: 0o755 …
�[90m |�[39m �[31m^�[39m
�[90m1592|�[39m const empty = await resolveQueue(REPOSITORY, SQUASH, deps);
�[90m1593|�[39m expect(empty.queue).toEqual([]);
�[90m �[2m❯�[22m processTicksAndRejections unknown:�[2m7:39�[22m�[39m
�[31m�[2m⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/2]⎯�[22m�[39m
�[41m�[1m FAIL �[22m�[49m test/review.test.ts�[2m > �[22ma reviewer's usage limit exits 3 with its reset, and records no round
�[31m�[1mError�[22m: Test timed out in 5000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".�[39m
�[36m �[2m❯�[22m <anonymous> test/review.test.ts:�[2m379:6�[22m�[39m
�[90m377|�[39m const round = review({ root: s.repo, signal: controller.signal }, { e…
�[90m378|�[39m while (!existsSync(pid)) await new Promise((resolve) => setTimeout(re…
�[90m379|�[39m controller.abort();
�[90m |�[39m �[31m^�[39m
�[90m380|�[39m await expect(round).rejects.toThrow(/abort/i);
�[90m381|�[39m // The round fails only once the CLI exited, so nothing outlives it i…
(written by codex / Codex) |
#497's own merge-group run failed on two root-suite tests while it changes only Markdown, and the same base passed in 122s on GitHub-hosted Ubuntu against 1590s on the queue runner. Rename the quest to match its wider goal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated
changed the title
There was a problem hiding this comment.
ℹ️ No critical issues — one small formatting nit inline, and the widened evidence all checks out.
Reviewed changes
Re-reviewed the two commits since the prior pullfrog review (f038009): the quest was widened from the Worker route tests to both suites on the queue runner and resized [M] → [L].
- Retitled the quest to
[L] Tests fit their timeout on the queue runner, widened its Goal to the root and Worker suites, and renamed the file toqueue-test-timeouts.md; the epic index row follows. - Rewrote the evidence block to "10 of 11 failed merge-queue runs failed in tests": Worker 7 runs (projects ×4, github ×3) plus root 3 runs, with run citations and the review/land/gate timeouts.
- Replaced the "22
checkjobs" lead with the count ofcheckjobs on main that finished that day and a cross-runner duration comparison (122s hosted vs 1590s queue on the same base, then 125s and 372s). - Added implementation leads for low-headroom tests (the
review.test.tsusage-limit case, #470's split) and theland.test.tsstall (1221s against its 30s limit,spawnSyncingit()).
I verified the new numbers directly: the 11 failed merge_group runs on 2026-11-04 via their failed logs (7 Worker, 3 root, and one that failed quest check on broken links instead of tests, so "10 of 11" is exact), the 1221s/30s land stall and 5917ms review timeout in run 41924229558, and the "27 check jobs" claim (25 successful runs plus 2 runs whose Ubuntu check job passed while Windows failed). Test names and spawnSync in src/cli/git.ts verify in the tree, and quest check --base origin/main passes.
glm-5.3-flash ｜ 𝕏
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Plans
quest/m0/queue-watcher/queue-test-timeouts.md, a follow-up of the land fixer's handback on moq-dev/quest#481. On 2026-11-04, 10 of 11 failed merge-queue runs failed in tests: 7 in Worker route tests (cloudflare/test/projects.test.tsfour times,cloudflare/test/github.test.tsthree times) and 3 in the root suite, including this PR's own run. Most were on PRs that didn't touch the failing code.Changes
[L] Tests fit their timeout on the queue runnerin m0's queue-watcher epic, ranked before log pruning.checkjob on main that finished that day (27) passed both suites on GitHub-hosted Ubuntu. On baseba352ea8, the root suite took 122s there (run 41924220424) while the queue runner took 1590s at the same time (run 41924229558), and 125s and 372s in the next two queue runs on that base.test/review.test.ts"a reviewer's usage limit exits 3 with its reset…", added by moq-dev/quest#471, runs four cases in one test and takes 3.3s of its 5s locally, right after moq-dev/quest#470 split that file to one variant per test for that reason.test/land.test.ts"opens the halt issue when the base fixer's launch was lost" stalled: 0.4s alone, but its 30s timeout fired only at 1221s, so its worker ran no timers for about 20 minutes.waitUntilbackground work, not a slow suite.Public API and wire impact: none, planning only.
Decisions
Settled with the maintainer in /quest-complete on 2026-11-04:
On 2026-11-04, after the land fixer handed this PR back (comment), the maintainer chose to add the new evidence before re-queueing, widening the Goal and title if it showed the root suite was affected too. It did, so
/quest-iteratemade these choices for review:[M]to[L].worker-test-timeouts.mdtoqueue-test-timeouts.md, so its path names its role. Nothing outside this PR links it.review.test.tstimeout, since the quest now covers it.Validation
just checkandjust testpass in the Nix dev shell on8e01e13e(vitest 1073 passed, 5 skipped; Worker 537 passed). On1005e897,just check,quest check, andquest check --base origin/mainpass.Review
Skipped on
1005e897f9f552e8de1e3240c21293f1548b4053as too trivial: its only change since the reviewed head rewraps one line of the quest file for Pullfrog's formatting nit, andgit diff -w --word-diffagainst8e01e13eshows no word changes.Codex
gpt-6.1-sol, effortxhigh, viaquest review --base origin/main, round 2 on8e01e13ed4df39b82bcbae38ea8b818e3e7e141b: no findings. Round 1 onf038009balso had none.Evidence checked against the run logs: per-test durations in run 41924229558, run 41923472254, run 41928750874, and run 41924220424; the failing tests of all 11 failed
merge_groupruns on 2026-11-04 viagh run view --log-failed; and three local runs each of the usage-limit test (3.3s) and the halt-issue test (0.4s).Follow-ups
None.
🤖 Generated with Claude Code
(Written by Claude Opus 5.5)