Repository navigation
Speed up test suite - #1356
Speed up test suite#1356
Conversation
The two floor tests each spawned tsc --listFilesOnly for the same deterministic count, and the end-to-end probe paid for it a third time inside the guard subprocess. Cache the count across the floor tests and let the guard accept it via DEAD_EXPORT_GUARD_SCANNED_FILES (test injection only; production never sets it), so the probe run pays only ts-prune.
The two large-file pagination chain tests paged a 45k-line (~11MB) fixture through 200-line offset hops. Paging is line-offset based: every hop re-reads the file from the start, so the cost is quadratic in fixture size (225 hops x up to 11MB each). The paging contract is ceiling-relative, so the tests now inject the 256KB scan ceiling (same as the single-line windowing test) and keep the fixture above it: 4k lines (~1MB) instead of 45k. Same contracts hold: no scan-limit trip on any hop, footer-driven continuation to the last line, exact row reassembly, and a deep page reachable past the ceiling. Production code is unchanged.
TheGreatAxios
left a comment
There was a problem hiding this comment.
Review · Approve
Shrinks the test-suite wall clock (~86s → ~45s) via test-injection seams (teardownDeadlineMs, maxScanBytes, DEAD_EXPORT_GUARD_SCANNED_FILES), smaller fixtures, and regenerated scripts/ci-timings.json; production behavior unchanged.
Findings
No findings.
Verification: diff stat is clean (no out-of-scope files, no Bin markers; touched prod files are exactly the three injection seams + the scan-limit message string), and all 7 commit subjects are typed test(...)/test: and ≤72 chars. Reviewer ran the targeted suites in the worktree: bun test src/plugins/read-file-guard-plugin.test.ts (41 pass — windowing/deep-page/contract checks intact), src/subagent/run-shell-child-reap.test.ts (5 pass — wedged-close settles in ~400ms via the 1s deadline), scripts/check-dead-exports.test.ts (32 pass, incl. "fails closed on a bogus value"), refresh-lock/session-refresh-race/test-parallel (16 pass), plus --randomize --seed 424242 runs (46 + 32 pass). Each seam defaults to the production constant when unset; DEAD_EXPORT_GUARD_SCANNED_FILES parsing rejects non-positive/non-integer values. ci-timings.json values track the measured per-file wall clocks with no contradiction.
Summary
This PR shrinks the test suite wall clock from ~86s to ~45s.
Each commit removes a fixed wall-clock cost from one slow test file:
Production behavior is unchanged; every change is a test-only injection or a smaller fixture. Pass/fail counts: 8334 pass, 4 skip, 0 fail before; 8336 pass, 4 skip, 0 fail after (two new injection tests added).
Metrics
Measured on this machine, seeded one-process suite:
bun test ./src ./e2e ./evals ./scripts ./testkit --randomize --seed 424242.faea4d4cinject short teardown deadlinesrc/subagent/run-shell-child-reap.test.ts70ec1fecinject scan ceilingsrc/plugins/read-file-guard-plugin.test.tsd5d460e8share tsc file-count runscripts/check-dead-exports.test.ts742e0435shrink chain fixturessrc/plugins/read-file-guard-plugin.test.tsNotes:
70ec1fecand742e0435both change
read-file-guard-plugin.test.ts(net 5.71s → 0.40s), andsuite wall clock varies by a few seconds under machine load.
scripts/check-dead-exports.test.ts(ts-prune/tsc subprocess) is the mainsource of run-to-run variance, not these changes.
showed 2 failures and 93.7s; every other run passes clean. Matches the
known flaky 15s timeout in
src/subagent/run-skill-scope.test.tsalreadylogged in downloads/README.md.
Commits
faea4d4cinject short teardown deadline in wedged-close testrunSubAgent's end-of-turn teardown on the non-persist path waited out theproduction 30s
DEFAULT_CLOSE_DEADLINE_MSbound whenagent.close()waswedged by a live shell descendant. The wedged-close test in
src/subagent/run-shell-child-reap.test.tstriggers that path, so it paidthe full 30s wall clock every run.
Added an optional
teardownDeadlineMsparam toRunSubAgentParamsthatoverrides that bound, following the existing test-injection pattern
(
outerRetryDelayMs,retryPolicy). Production never sets it and keeps the30s default, so runtime behavior is unchanged. The test passes a 1s deadline
and still exercises the same contract: the bounded teardown fires on the
wedged close and settles the run.
Regenerated
scripts/ci-timings.jsonper the documented policy (slow filechanged) so CI shard balancing uses the new per-file durations.
70ec1fecinject scan ceiling to shrink fixtureThe single-line pagination test in
src/plugins/read-file-guard-plugin.test.tsbuilt an 8MB fixture (
READ_FILE_MAX_SCAN_BYTES + 4096) and paged through itwith repeated
readFileBoundedcalls. Paging is stateless and line-offsetbased: every hop opens a fresh stream at the start of the file and re-reads
all bytes up to its offset, so an 8MB one-line fixture cost ~168 hops, each
re-reading up to 8MB (~4s of the 5.7s file; quadratic in fixture size).
Added an optional
maxScanBytestoReadFileGuardPluginOptionsthatoverrides the scan ceiling, following the existing test-injection pattern
(
whichExtractor,canExecuteHostCommands,teardownDeadlineMs).Production never sets it and keeps the 8MB
READ_FILE_MAX_SCAN_BYTESdefault, so runtime behavior is unchanged. The scan-limit messages now
report the effective ceiling instead of the constant. The test injects a
256KB ceiling and keeps its fixture at ceiling+4096: same contract (single
line past the scan ceiling windowed across hops with exact reassembly, no
silent truncation, no scan-limit footer), ~6 hops over a 260KB file.
Regenerated
scripts/ci-timings.jsonper the documented policy (slow filechanged) so CI shard balancing uses the new per-file durations.
d5d460e8share tsc file-count run and inject it in the probescripts/check-dead-exports.test.tswas the largest file in the suite(~9.3s, ~18%). Its cost came from running
tsc --listFilesOnlythree timesfor the same deterministic count: once per floor test (~1.45s each) and a
third time inside the guard subprocess spawned by the end-to-end probe
(ts-prune ~4.6s + tsc ~1.5s).
cache (
liveScannedFileCount); each test still computes it on first use,so no test depends on another having run.
scripts/check-dead-exports.ts) accepts the count via theDEAD_EXPORT_GUARD_SCANNED_FILESenv var (test injection only, samepattern as
teardownDeadlineMs/maxScanBytes; production never sets it,so the gate always computes the count in normal runs). The end-to-end
probe passes the cached real count, so the subprocess pays only ts-prune
while still exercising the full gate wiring: config load, allowlist
validation, real ts-prune over the real program, floor check, and exit 1
naming the probe.
integer pass, bogus values fail closed).
Regenerated
scripts/ci-timings.jsonper the documented policy (slow filechanged) so CI shard balancing uses the new per-file durations.
742e0435shrink chain fixtures via scan-ceiling injectionThe two large-file pagination chain tests in
src/plugins/read-file-guard-plugin.test.tspaged a 45k-line (~11MB)fixture through 200-line offset hops. Paging is line-offset based and
stateless: every hop opens a fresh stream at the start of the file and
re-reads all bytes up to its offset, so the cost is quadratic in fixture
size (225 hops x up to 11MB each, ~0.85s of the file's 1.27s).
The paging contract is ceiling-relative, so the chain tests now inject the
256KB scan ceiling (same value the single-line windowing test uses) and
keep the fixture above it: 4k lines (~1MB) instead of 45k. Same contracts
hold: no scan-limit trip on any hop, footer-driven continuation to the last
line, exact row reassembly, and a deep page (line 3500 of 4000) reachable
past the ceiling. The deep-page test passes the injection too, so its
"larger than the scan ceiling" property holds relative to the effective
ceiling. Production code is unchanged; the injection point already existed
from the prior commit.
Regenerated
scripts/ci-timings.jsonper the documented policy (slow filechanged) so CI shard balancing uses the new per-file durations. Four-shard
balance is unchanged (~11.3s per shard).