diff --git a/quest/m0/queue-watcher/README.md b/quest/m0/queue-watcher/README.md index 1dab324b..071c6c1e 100644 --- a/quest/m0/queue-watcher/README.md +++ b/quest/m0/queue-watcher/README.md @@ -137,6 +137,7 @@ Shared shape, for the children to follow: - [The other repositories follow](/quest/m0/queue-watcher/follow.md) - quest, rind and rust2 are cut over and landing; waits for a fork PR to land through one of their queues - [Harden the queue runners](/quest/m0/queue-watcher/runners.md) - each queue runner runs only its repository's queue workflow, and every outside fork PR needs approval +- [Tests fit their timeout on the queue runner](/quest/m0/queue-watcher/queue-test-timeouts.md) - the root and Worker suites stop timing out or stalling in merge-group runs, so the queue stops ejecting PRs for them - [Daemons prune their logs](/quest/m0/queue-watcher/log-retention.md) - the watcher and gate delete their finished logs after a window, so state stays bounded (4.1 GiB on nixos on 2026-11-04) - [Retire the batching gate](/quest/m0/queue-watcher/retire.md) - once no repository lands through `quest land serve`, its batching code and host units go diff --git a/quest/m0/queue-watcher/queue-test-timeouts.md b/quest/m0/queue-watcher/queue-test-timeouts.md new file mode 100644 index 00000000..72255c75 --- /dev/null +++ b/quest/m0/queue-watcher/queue-test-timeouts.md @@ -0,0 +1,100 @@ +# [L] Tests fit their timeout on the queue runner + +## Goal + +The root and Worker test suites pass in `merge_group` runs on the queue +runner without timing out or stalling, so the merge queue stops ejecting PRs +for tests they don't touch. Each test fits comfortably under its limit for a +measured reason, rather than through a blanket timeout raise. + +## Plan + +Decided 2026-11-04 (user, /quest-complete follow-up of +[moq-dev/quest#481](https://github.com/moq-dev/quest/pull/481)), on the Worker +route tests alone. Widened to the root suite the same day, when this quest's +own PR, [moq-dev/quest#497](https://github.com/moq-dev/quest/pull/497), was +ejected for two root-suite tests although it changes only Markdown. + +That day, 10 of 11 failed merge-queue runs failed in tests: + +- Worker, 7 runs: `cloudflare/test/projects.test.ts` ("enrolled project + observations", four times) and `cloudflare/test/github.test.ts` ("signed + GitHub webhooks", three times) exceeded vitest's 5s default, six of them on + PRs that didn't touch the Worker + ([run 41901538568](https://github.com/moq-dev/quest/actions/runs/41901538568) + is one). +- Root, 3 runs: `test/review.test.ts` timed out twice, "a missing or + logged-out reviewer CLI is a usage error" at 5.2s + ([run 41905766925](https://github.com/moq-dev/quest/actions/runs/41905766925)) + and "a reviewer's usage limit exits 3 with its reset, and records no round" + at 5.9s; `test/land.test.ts` "host daemon > opens the halt issue when the + base fixer's launch was lost" ran 1221s against its 30s limit, both in + [run 41924229558](https://github.com/moq-dev/quest/actions/runs/41924229558); + and `test/gate.test.ts` "killing stops a command that ignores the request" + failed an assertion + ([run 41888531939](https://github.com/moq-dev/quest/actions/runs/41888531939)). + +The land fixer re-queued each as flaky or handed it back, costing a queue +slot each time; #481 hit its three-flaky stop rule. + +- What should the quest's goal be? + - ✅ Fix the root cause: find what makes these tests slow in merge-group + runs and make each fit, raising a timeout only where a measured cost + justifies it + - Rejected: raising the suite's timeout; leaving it to the watcher's flaky + handling +- Which milestone? + - ✅ m0, in this epic, since it stalls the merge queue today + - Rejected: m1 + +Implementation guidance: + +- It looks environmental, and specific to the queue runner. The queue build + runs on `nixos-quest-ci`, which + [kixelated/nixos#19](https://github.com/kixelated/nixos/pull/19) added to + the maintainer's host + ([the cutover](/quest/m0/queue-watcher/follow.md)); + [Harden the queue runners](/quest/m0/queue-watcher/runners.md) only + restricts its access. Every `check` job of the `Check` workflow on main + that finished that day, 27 by 23:00 UTC, passed both suites on + GitHub-hosted Ubuntu. On base `ba352ea8`, the + root suite took 122s there + ([run 41924220424](https://github.com/moq-dev/quest/actions/runs/41924220424)) + while the queue runner took 1590s on it at the same time (run 41924229558), + and 125s and 372s in the next queue runs on the same base + ([run 41923472254](https://github.com/moq-dev/quest/actions/runs/41923472254), + [run 41928750874](https://github.com/moq-dev/quest/actions/runs/41928750874)). + In the slow run, `land.test.ts` tests that took 0.5-2.6s in the other two + took 12-23s. Measure the host's load and memory pressure during queue builds, + and what else runs there, before blaming the tests. +- Some tests have little headroom even on an idle host. + "a reviewer's usage limit exits 3 with its reset, and records no round", + added by [moq-dev/quest#471](https://github.com/moq-dev/quest/pull/471), + runs four cases in one test, each with its own repository and two `quest` + CLI runs: 3.3s of its 5s on a developer machine. + [moq-dev/quest#470](https://github.com/moq-dev/quest/pull/470) had just + split this file's tests to one `quest review` per variant for that reason. + Find such tests by their share of the limit on a + quiet host, not by which one timed out last. +- The `land.test.ts` run is a stall, not slowness: the test takes 0.4s alone, + and its 30s timeout fired only at 1221s, so its vitest worker ran no timers + for about 20 minutes. Either a synchronous call blocked it, such as the + `spawnSync` in `git()` (`src/cli/git.ts`), or the host didn't schedule it. +- The Worker timeouts look like one test stalling, not a slow suite: in + [run 41886648526](https://github.com/moq-dev/quest/actions/runs/41886648526) + no other Worker test took over 807ms. Every timed-out test delivers an + admitted `pull_request` or `push` webhook, whose route starts `waitUntil` + background work (`cloudflare/src/github/webhook.ts`), and in + [run 41879887537](https://github.com/moq-dev/quest/actions/runs/41879887537) + the next test failed 396ms after the timeout, as if that work outlived its + test under `isolate: false`. `github.test.ts` stubs no global `fetch`, so its + background reads may reach the network. +- Earlier rounds: [moq-dev/quest#112](https://github.com/moq-dev/quest/pull/112) + shared one workerd across files, and + [moq-dev/quest#130](https://github.com/moq-dev/quest/pull/130) moved + Miniflare's D1 to `/dev/shm`, both in `cloudflare/vitest.config.ts`. + [moq-dev/quest#177](https://github.com/moq-dev/quest/pull/177)'s + `scripts/gate-bench.sh` reproduces suite contention, and + `just worker-test-isolated` runs each Worker case alone. +- Verify with repeated runs under the queue runner's conditions, not one + green run.