From 9de3cc59f7f5054c571d7e0463f677b630c1edc9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 13:43:18 -0800 Subject: [PATCH 1/4] quest: plan fixing the Worker route test timeouts in the merge queue Co-Authored-By: Claude Opus 5.5 --- quest/m0/queue-watcher/README.md | 1 + .../m0/queue-watcher/worker-test-timeouts.md | 41 +++++++++++++++++++ 2 files changed, 42 insertions(+) create mode 100644 quest/m0/queue-watcher/worker-test-timeouts.md diff --git a/quest/m0/queue-watcher/README.md b/quest/m0/queue-watcher/README.md index 1dab324b..0158c411 100644 --- a/quest/m0/queue-watcher/README.md +++ b/quest/m0/queue-watcher/README.md @@ -137,6 +137,7 @@ Shared shape, for the children to follow: - [The other repositories follow](/quest/m0/queue-watcher/follow.md) - quest, rind and rust2 are cut over and landing; waits for a fork PR to land through one of their queues - [Harden the queue runners](/quest/m0/queue-watcher/runners.md) - each queue runner runs only its repository's queue workflow, and every outside fork PR needs approval +- [Worker route tests fit their timeout](/quest/m0/queue-watcher/worker-test-timeouts.md) - the Worker's route tests stop timing out in merge-group runs, so the queue stops ejecting PRs for them - [Daemons prune their logs](/quest/m0/queue-watcher/log-retention.md) - the watcher and gate delete their finished logs after a window, so state stays bounded (4.1 GiB on nixos on 2026-11-04) - [Retire the batching gate](/quest/m0/queue-watcher/retire.md) - once no repository lands through `quest land serve`, its batching code and host units go diff --git a/quest/m0/queue-watcher/worker-test-timeouts.md b/quest/m0/queue-watcher/worker-test-timeouts.md new file mode 100644 index 00000000..15af7482 --- /dev/null +++ b/quest/m0/queue-watcher/worker-test-timeouts.md @@ -0,0 +1,41 @@ +# [M] Worker route tests fit their timeout + +## Goal + +The Worker's route tests pass in `merge_group` runs without timing out, so +the merge queue stops ejecting PRs for them. Each test fits comfortably under +its limit for a measured reason, rather than through a blanket timeout raise. + +## Plan + +Decided 2026-11-04 (user, /quest-complete follow-up of +[moq-dev/quest#481](https://github.com/moq-dev/quest/pull/481)). That day, +5 of 6 failed merge-queue runs were Worker route tests exceeding vitest's 5s +default: `cloudflare/test/projects.test.ts` ("enrolled project observations", +three times) and `cloudflare/test/github.test.ts` ("signed GitHub webhooks", +twice), on PRs that didn't touch the Worker +([run 41901538568](https://github.com/moq-dev/quest/actions/runs/41901538568) +is one). The land fixer re-queued each as flaky, costing a queue slot each +time, and #481 hit its three-flaky stop rule. + +- What should the quest's goal be? + - ✅ Fix the root cause: find what makes these tests slow in merge-group + runs and make each fit, raising a timeout only where a measured cost + justifies it + - Rejected: raising the suite's timeout; leaving it to the watcher's flaky + handling +- Which milestone? + - ✅ m0, in this epic, since it stalls the merge queue today + - Rejected: m1 + +Implementation guidance: + +- Measure first: per-test durations on the queue runners against a local + run, and where the time goes (Miniflare and D1 setup per test, the auth + flows, signing, host load). `cloudflare/vitest.config.ts` already moved + Miniflare's D1 to `/dev/shm` for a similar stall. +- Check whether the queue runners share a loaded host + ([Harden the queue runners](/quest/m0/queue-watcher/runners.md) owns + their setup) before blaming the tests. +- Verify with repeated runs under the queue runner's conditions, not one + green run. From f038009bd5fa6526a96541c700d47a52e9f11ce8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 13:49:07 -0800 Subject: [PATCH 2/4] quest: correct the timeout counts and point the plan at the runner and background work Co-Authored-By: Claude Opus 5.5 --- .../m0/queue-watcher/worker-test-timeouts.md | 41 ++++++++++++++----- 1 file changed, 31 insertions(+), 10 deletions(-) diff --git a/quest/m0/queue-watcher/worker-test-timeouts.md b/quest/m0/queue-watcher/worker-test-timeouts.md index 15af7482..24e8c9ad 100644 --- a/quest/m0/queue-watcher/worker-test-timeouts.md +++ b/quest/m0/queue-watcher/worker-test-timeouts.md @@ -10,10 +10,10 @@ its limit for a measured reason, rather than through a blanket timeout raise. Decided 2026-11-04 (user, /quest-complete follow-up of [moq-dev/quest#481](https://github.com/moq-dev/quest/pull/481)). That day, -5 of 6 failed merge-queue runs were Worker route tests exceeding vitest's 5s +7 of 10 failed merge-queue runs were Worker route tests exceeding vitest's 5s default: `cloudflare/test/projects.test.ts` ("enrolled project observations", -three times) and `cloudflare/test/github.test.ts` ("signed GitHub webhooks", -twice), on PRs that didn't touch the Worker +four times) and `cloudflare/test/github.test.ts` ("signed GitHub webhooks", +three times), six of them on PRs that didn't touch the Worker ([run 41901538568](https://github.com/moq-dev/quest/actions/runs/41901538568) is one). The land fixer re-queued each as flaky, costing a queue slot each time, and #481 hit its three-flaky stop rule. @@ -30,12 +30,33 @@ time, and #481 hit its three-flaky stop rule. Implementation guidance: -- Measure first: per-test durations on the queue runners against a local - run, and where the time goes (Miniflare and D1 setup per test, the auth - flows, signing, host load). `cloudflare/vitest.config.ts` already moved - Miniflare's D1 to `/dev/shm` for a similar stall. -- Check whether the queue runners share a loaded host - ([Harden the queue runners](/quest/m0/queue-watcher/runners.md) owns - their setup) before blaming the tests. +- It looks environmental: the same day, all 22 `check` jobs of the `Check` + workflow on main ran the whole Worker suite on GitHub-hosted Ubuntu + without a timeout. The queue build runs on `nixos-quest-ci`, which + [kixelated/nixos#19](https://github.com/kixelated/nixos/pull/19) added to + the maintainer's host + ([the cutover](/quest/m0/queue-watcher/follow.md)); + [Harden the queue runners](/quest/m0/queue-watcher/runners.md) only + restricts its access. A root-suite test, `test/review.test.ts` "a missing or + logged-out reviewer CLI is a usage error", also timed out there at 5.2s + ([run 41905766925](https://github.com/moq-dev/quest/actions/runs/41905766925)), + so measure the host's load and memory pressure during queue builds before + blaming the tests. +- It looks like a stall in one test, not a slow suite: in + [run 41886648526](https://github.com/moq-dev/quest/actions/runs/41886648526) + no other Worker test took over 807ms. Every timed-out test delivers an + admitted `pull_request` or `push` webhook, whose route starts `waitUntil` + background work (`cloudflare/src/github/webhook.ts`), and in + [run 41879887537](https://github.com/moq-dev/quest/actions/runs/41879887537) + the next test failed 396ms after the timeout, as if that work outlived its + test under `isolate: false`. `github.test.ts` stubs no global `fetch`, so its + background reads may reach the network. +- Earlier rounds: [moq-dev/quest#112](https://github.com/moq-dev/quest/pull/112) + shared one workerd across files, and + [moq-dev/quest#130](https://github.com/moq-dev/quest/pull/130) moved + Miniflare's D1 to `/dev/shm`, both in `cloudflare/vitest.config.ts`. + [moq-dev/quest#177](https://github.com/moq-dev/quest/pull/177)'s + `scripts/gate-bench.sh` reproduces suite contention, and + `just worker-test-isolated` runs each case alone. - Verify with repeated runs under the queue runner's conditions, not one green run. From 8e01e13ed4df39b82bcbae38ea8b818e3e7e141b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 15:06:34 -0800 Subject: [PATCH 3/4] quest: widen the timeout quest to the root suite on the queue runner #497's own merge-group run failed on two root-suite tests while it changes only Markdown, and the same base passed in 122s on GitHub-hosted Ubuntu against 1590s on the queue runner. Rename the quest to match its wider goal. Co-Authored-By: Claude Opus 5.5 --- quest/m0/queue-watcher/README.md | 2 +- quest/m0/queue-watcher/queue-test-timeouts.md | 99 +++++++++++++++++++ .../m0/queue-watcher/worker-test-timeouts.md | 62 ------------ 3 files changed, 100 insertions(+), 63 deletions(-) create mode 100644 quest/m0/queue-watcher/queue-test-timeouts.md delete mode 100644 quest/m0/queue-watcher/worker-test-timeouts.md diff --git a/quest/m0/queue-watcher/README.md b/quest/m0/queue-watcher/README.md index 0158c411..071c6c1e 100644 --- a/quest/m0/queue-watcher/README.md +++ b/quest/m0/queue-watcher/README.md @@ -137,7 +137,7 @@ Shared shape, for the children to follow: - [The other repositories follow](/quest/m0/queue-watcher/follow.md) - quest, rind and rust2 are cut over and landing; waits for a fork PR to land through one of their queues - [Harden the queue runners](/quest/m0/queue-watcher/runners.md) - each queue runner runs only its repository's queue workflow, and every outside fork PR needs approval -- [Worker route tests fit their timeout](/quest/m0/queue-watcher/worker-test-timeouts.md) - the Worker's route tests stop timing out in merge-group runs, so the queue stops ejecting PRs for them +- [Tests fit their timeout on the queue runner](/quest/m0/queue-watcher/queue-test-timeouts.md) - the root and Worker suites stop timing out or stalling in merge-group runs, so the queue stops ejecting PRs for them - [Daemons prune their logs](/quest/m0/queue-watcher/log-retention.md) - the watcher and gate delete their finished logs after a window, so state stays bounded (4.1 GiB on nixos on 2026-11-04) - [Retire the batching gate](/quest/m0/queue-watcher/retire.md) - once no repository lands through `quest land serve`, its batching code and host units go diff --git a/quest/m0/queue-watcher/queue-test-timeouts.md b/quest/m0/queue-watcher/queue-test-timeouts.md new file mode 100644 index 00000000..61982ef5 --- /dev/null +++ b/quest/m0/queue-watcher/queue-test-timeouts.md @@ -0,0 +1,99 @@ +# [L] Tests fit their timeout on the queue runner + +## Goal + +The root and Worker test suites pass in `merge_group` runs on the queue +runner without timing out or stalling, so the merge queue stops ejecting PRs +for tests they don't touch. Each test fits comfortably under its limit for a +measured reason, rather than through a blanket timeout raise. + +## Plan + +Decided 2026-11-04 (user, /quest-complete follow-up of +[moq-dev/quest#481](https://github.com/moq-dev/quest/pull/481)), on the Worker +route tests alone. Widened to the root suite the same day, when this quest's +own PR, [moq-dev/quest#497](https://github.com/moq-dev/quest/pull/497), was +ejected for two root-suite tests although it changes only Markdown. + +That day, 10 of 11 failed merge-queue runs failed in tests: + +- Worker, 7 runs: `cloudflare/test/projects.test.ts` ("enrolled project + observations", four times) and `cloudflare/test/github.test.ts` ("signed + GitHub webhooks", three times) exceeded vitest's 5s default, six of them on + PRs that didn't touch the Worker + ([run 41901538568](https://github.com/moq-dev/quest/actions/runs/41901538568) + is one). +- Root, 3 runs: `test/review.test.ts` timed out twice, "a missing or + logged-out reviewer CLI is a usage error" at 5.2s + ([run 41905766925](https://github.com/moq-dev/quest/actions/runs/41905766925)) + and "a reviewer's usage limit exits 3 with its reset, and records no round" + at 5.9s; `test/land.test.ts` "host daemon > opens the halt issue when the + base fixer's launch was lost" ran 1221s against its 30s limit, both in + [run 41924229558](https://github.com/moq-dev/quest/actions/runs/41924229558); + and `test/gate.test.ts` "killing stops a command that ignores the request" + failed an assertion + ([run 41888531939](https://github.com/moq-dev/quest/actions/runs/41888531939)). + +The land fixer re-queued each as flaky or handed it back, costing a queue +slot each time; #481 hit its three-flaky stop rule. + +- What should the quest's goal be? + - ✅ Fix the root cause: find what makes these tests slow in merge-group + runs and make each fit, raising a timeout only where a measured cost + justifies it + - Rejected: raising the suite's timeout; leaving it to the watcher's flaky + handling +- Which milestone? + - ✅ m0, in this epic, since it stalls the merge queue today + - Rejected: m1 + +Implementation guidance: + +- It looks environmental, and specific to the queue runner. The queue build + runs on `nixos-quest-ci`, which + [kixelated/nixos#19](https://github.com/kixelated/nixos/pull/19) added to + the maintainer's host + ([the cutover](/quest/m0/queue-watcher/follow.md)); + [Harden the queue runners](/quest/m0/queue-watcher/runners.md) only + restricts its access. Every `check` job of the `Check` workflow on main + that finished that day, 27 by 23:00 UTC, passed both suites on + GitHub-hosted Ubuntu. On base `ba352ea8`, the + root suite took 122s there + ([run 41924220424](https://github.com/moq-dev/quest/actions/runs/41924220424)) + while the queue runner took 1590s on it at the same time (run 41924229558), + and 125s and 372s in the next queue runs on the same base + ([run 41923472254](https://github.com/moq-dev/quest/actions/runs/41923472254), + [run 41928750874](https://github.com/moq-dev/quest/actions/runs/41928750874)). + In the slow run, `land.test.ts` tests that took 0.5-2.6s in the other two + took 12-23s. Measure the host's load and memory pressure during queue builds, + and what else runs there, before blaming the tests. +- Some tests have little headroom even on an idle host. + "a reviewer's usage limit exits 3 with its reset, and records no round", + added by [moq-dev/quest#471](https://github.com/moq-dev/quest/pull/471), + runs four cases in one test, each with its own repository and two `quest` + CLI runs: 3.3s of its 5s on a developer machine. + [moq-dev/quest#470](https://github.com/moq-dev/quest/pull/470) had just + split this file's tests to one `quest review` per variant for that reason. Find such tests by their share of the limit on a + quiet host, not by which one timed out last. +- The `land.test.ts` run is a stall, not slowness: the test takes 0.4s alone, + and its 30s timeout fired only at 1221s, so its vitest worker ran no timers + for about 20 minutes. Either a synchronous call blocked it, such as the + `spawnSync` in `git()` (`src/cli/git.ts`), or the host didn't schedule it. +- The Worker timeouts look like one test stalling, not a slow suite: in + [run 41886648526](https://github.com/moq-dev/quest/actions/runs/41886648526) + no other Worker test took over 807ms. Every timed-out test delivers an + admitted `pull_request` or `push` webhook, whose route starts `waitUntil` + background work (`cloudflare/src/github/webhook.ts`), and in + [run 41879887537](https://github.com/moq-dev/quest/actions/runs/41879887537) + the next test failed 396ms after the timeout, as if that work outlived its + test under `isolate: false`. `github.test.ts` stubs no global `fetch`, so its + background reads may reach the network. +- Earlier rounds: [moq-dev/quest#112](https://github.com/moq-dev/quest/pull/112) + shared one workerd across files, and + [moq-dev/quest#130](https://github.com/moq-dev/quest/pull/130) moved + Miniflare's D1 to `/dev/shm`, both in `cloudflare/vitest.config.ts`. + [moq-dev/quest#177](https://github.com/moq-dev/quest/pull/177)'s + `scripts/gate-bench.sh` reproduces suite contention, and + `just worker-test-isolated` runs each Worker case alone. +- Verify with repeated runs under the queue runner's conditions, not one + green run. diff --git a/quest/m0/queue-watcher/worker-test-timeouts.md b/quest/m0/queue-watcher/worker-test-timeouts.md deleted file mode 100644 index 24e8c9ad..00000000 --- a/quest/m0/queue-watcher/worker-test-timeouts.md +++ /dev/null @@ -1,62 +0,0 @@ -# [M] Worker route tests fit their timeout - -## Goal - -The Worker's route tests pass in `merge_group` runs without timing out, so -the merge queue stops ejecting PRs for them. Each test fits comfortably under -its limit for a measured reason, rather than through a blanket timeout raise. - -## Plan - -Decided 2026-11-04 (user, /quest-complete follow-up of -[moq-dev/quest#481](https://github.com/moq-dev/quest/pull/481)). That day, -7 of 10 failed merge-queue runs were Worker route tests exceeding vitest's 5s -default: `cloudflare/test/projects.test.ts` ("enrolled project observations", -four times) and `cloudflare/test/github.test.ts` ("signed GitHub webhooks", -three times), six of them on PRs that didn't touch the Worker -([run 41901538568](https://github.com/moq-dev/quest/actions/runs/41901538568) -is one). The land fixer re-queued each as flaky, costing a queue slot each -time, and #481 hit its three-flaky stop rule. - -- What should the quest's goal be? - - ✅ Fix the root cause: find what makes these tests slow in merge-group - runs and make each fit, raising a timeout only where a measured cost - justifies it - - Rejected: raising the suite's timeout; leaving it to the watcher's flaky - handling -- Which milestone? - - ✅ m0, in this epic, since it stalls the merge queue today - - Rejected: m1 - -Implementation guidance: - -- It looks environmental: the same day, all 22 `check` jobs of the `Check` - workflow on main ran the whole Worker suite on GitHub-hosted Ubuntu - without a timeout. The queue build runs on `nixos-quest-ci`, which - [kixelated/nixos#19](https://github.com/kixelated/nixos/pull/19) added to - the maintainer's host - ([the cutover](/quest/m0/queue-watcher/follow.md)); - [Harden the queue runners](/quest/m0/queue-watcher/runners.md) only - restricts its access. A root-suite test, `test/review.test.ts` "a missing or - logged-out reviewer CLI is a usage error", also timed out there at 5.2s - ([run 41905766925](https://github.com/moq-dev/quest/actions/runs/41905766925)), - so measure the host's load and memory pressure during queue builds before - blaming the tests. -- It looks like a stall in one test, not a slow suite: in - [run 41886648526](https://github.com/moq-dev/quest/actions/runs/41886648526) - no other Worker test took over 807ms. Every timed-out test delivers an - admitted `pull_request` or `push` webhook, whose route starts `waitUntil` - background work (`cloudflare/src/github/webhook.ts`), and in - [run 41879887537](https://github.com/moq-dev/quest/actions/runs/41879887537) - the next test failed 396ms after the timeout, as if that work outlived its - test under `isolate: false`. `github.test.ts` stubs no global `fetch`, so its - background reads may reach the network. -- Earlier rounds: [moq-dev/quest#112](https://github.com/moq-dev/quest/pull/112) - shared one workerd across files, and - [moq-dev/quest#130](https://github.com/moq-dev/quest/pull/130) moved - Miniflare's D1 to `/dev/shm`, both in `cloudflare/vitest.config.ts`. - [moq-dev/quest#177](https://github.com/moq-dev/quest/pull/177)'s - `scripts/gate-bench.sh` reproduces suite contention, and - `just worker-test-isolated` runs each case alone. -- Verify with repeated runs under the queue runner's conditions, not one - green run. From 1005e897f9f552e8de1e3240c21293f1548b4053 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 15:50:29 -0800 Subject: [PATCH 4/4] quest: rewrap the overlong headroom line in the timeout quest Co-Authored-By: Claude Opus 5.5 --- quest/m0/queue-watcher/queue-test-timeouts.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/quest/m0/queue-watcher/queue-test-timeouts.md b/quest/m0/queue-watcher/queue-test-timeouts.md index 61982ef5..72255c75 100644 --- a/quest/m0/queue-watcher/queue-test-timeouts.md +++ b/quest/m0/queue-watcher/queue-test-timeouts.md @@ -73,7 +73,8 @@ Implementation guidance: runs four cases in one test, each with its own repository and two `quest` CLI runs: 3.3s of its 5s on a developer machine. [moq-dev/quest#470](https://github.com/moq-dev/quest/pull/470) had just - split this file's tests to one `quest review` per variant for that reason. Find such tests by their share of the limit on a + split this file's tests to one `quest review` per variant for that reason. + Find such tests by their share of the limit on a quiet host, not by which one timed out last. - The `land.test.ts` run is a stall, not slowness: the test takes 0.4s alone, and its 30s timeout fired only at 1221s, so its vitest worker ran no timers