Skip to content

test: time out on CPU time, not wall time - #631

Merged
kixelated merged 8 commits into
mainfrom
quest/m0/test-cpu-timeout
Nov 11, 2026
Merged

kixelated merged 8 commits into
mainfrom
quest/m0/test-cpu-timeout

Conversation

@kixelated

@kixelated kixelated commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m0/test-cpu-timeout.md.

Summary

The merge-group build kept dropping PRs for Test timed out in 5000ms (or a 20 s / 30 s override) on the shared self-hosted runner: a wall-clock limit measured the host's load, not the tests.

  • test/budget.ts, a vitest setup file, fails a test that spends more than 10 s of CPU, counting its hooks and every child process it waited for. On Linux it reads the worker's utime + stime + cutime + cstime from /proc/self/stat before and after each test.
  • vitest.config.ts raises testTimeout and hookTimeout to 5 minutes, which only catches a hang (a hung test waiting on a child or IO spends no CPU). The live suite keeps its own limits and no CPU budget.
  • Other platforms decide at setup: the budget registers only on Linux, so macOS and Windows use the wall bound alone.
  • Removed every per-test wall override (vi.setConfig({ testTimeout }), { timeout } on describes and tests, trailing }, 60_000)), each of which existed only to survive load or slow Windows process creation.
  • Polled waits lost their fixed wall deadlines, which also failed under load: until in test/gate.test.ts (200 x 25 ms, seen failing at 48-way load in moq-dev/quest#629), the matching loop in test/worktree.run.test.ts, and the 5 s loops in test/land-fixer-diagnostics.test.ts. A wait for a file now uses the fixture's created watcher; a wait for a process to stop still polls, bounded by the test's wall limit.
  • Deleted the quest.

Measurements

Full suite on the maintainer's host (24 CPUs, load average about 40), 2026-11-11: 1222 tests, 128.6 s of CPU in total; the most any single test spent was 1.9 s (review.test.ts "the PR's Models record sizes the review..."), then 1.6 s and 1.5 s in land.test.ts. The 10 s budget is about five times that peak, leaving room for slower CI cores and contention.

A throwaway probe confirmed the behavior: a test spinning an 11 s child and one spinning itself for 11 s both failed with test spent 10990 ms of CPU, over its 10000 ms budget, while a test waiting 6 s without CPU (past the old 5 s default) passed.

Choices made without asking

  • CPU budget: one global 10 s budget rather than per-test budgets, since no test comes near it.
  • The budget is checked in afterEach, so a CPU-bound test fails once it finishes; one that never finishes fails at the 5-minute wall bound. Vitest offers no way to fail a running test from outside.
  • test/gate.test.ts lost its { timeout: 30_000 } options, and prettier refolded the multi-line test(...) calls they had forced.

Decisions

  • How to measure a test's CPU (the user, 2026-11-11): ✅ the worker's own and reaped children's CPU from /proc/self/stat (rejected: delegated per-worker cgroups). The gate's scope wraps the whole just _test, so its cpu.stat cannot attribute CPU to one test, and a per-worker cgroup would miss the commands the gate and worktree tests move into their own systemd scopes, which cutime still counts. The CPU budget applies in addition to a wall-clock bound that catches hangs (the 5-minute limit here); that bound also covers a descendant the test never waits for, which /proc/self/stat does not count. This replaces the quest's earlier "cgroup cpu.stat" plan.
  • Wall bound (the user, 2026-11-11): ✅ keep a wall bound that catches hangs, "as the PR's 5-minute limit does", for tests and hooks (rejected: a CPU budget alone). The same brief directed this PR to remove until's fixed 200 x 25 ms deadline in test/gate.test.ts, so polled waits carry no deadline of their own: a wait that never completes fails its test at the 5-minute bound. A regression that hangs many waits in one file can therefore take the queue build to its 60-minute job limit before the file reports; a run of that file locally names the stalled tests.

Public API

None. Test configuration only; no CLI, wire, or on-disk change.

Caused by

No single PR caused this; vitest's wall-clock default is as old as the suite.

Validation

  • just check: passed.
  • just test (vitest, install and lifecycle checks, under quest gate): passed, 1239 tests, after merging main (with moq-dev/quest#629).

Follow-ups

None.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, medium effort
Implementer: Claude Code, claude-opus-5-5, medium effort (iterate, M)

(Written by Claude Opus 5.5 with medium)

Review

Rewritten by quest review after each round.

  • Head: 4d4e7d8

  • Base: origin/main at 9bcfd22

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1cbdc-a8e6-7121-86bd-c29506759a8c, round 4

  • Score: 10/10, The incremental changes merge the updated base without introducing new branch-specific code or review findings.

  • Findings: none

  • Reviewer: opencode, opencode-go/deepseek-v4.1-flash, high effort

  • Session: ses_e34235094ffel7VB11tUcB2AIn, round 4

  • Score: 10/10, The merge with origin/main is clean: both conflicted quest files end deleted with no dangling references, and the branch's budget, config, and test edits survive intact, so nothing new holds at this head.

  • Findings: none

kixelated and others added 3 commits November 11, 2026 00:54
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Tests time out on CPU time, not wall time test: time out on CPU time, not wall time Nov 11, 2026
kixelated and others added 4 commits November 11, 2026 10:24
The test's wall bound catches a wait that never ends; waits for a file use
created's watcher instead of a poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…und catches

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 11, 2026 19:24
@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Pre-merge checks: Inconclusive 2

Failed Failed checks: 2 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Description check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…eout

# Conflicts:
#	quest/m0/gate-test-races.md
#	quest/m0/test-cpu-timeout.md
@kixelated
kixelated added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit a8fa332 Nov 11, 2026
7 of 8 checks passed
@kixelated
kixelated deleted the quest/m0/test-cpu-timeout branch November 11, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant