Skip to content

Speed up test suite - #1356

Merged
Gustav-Simonsson merged 7 commits into
corbitsdev:mainfrom
Gustav-Simonsson:faster_tests2
Oct 7, 2026
Merged

Gustav-Simonsson merged 7 commits into
corbitsdev:mainfrom
Gustav-Simonsson:faster_tests2

Conversation

@Gustav-Simonsson

Copy link
Copy Markdown
Contributor

Summary

This PR shrinks the test suite wall clock from ~86s to ~45s.

Each commit removes a fixed wall-clock cost from one slow test file:

  • a short teardown deadline for the wedged-close subagent test
  • a scan ceiling for the read-file guard tests
  • a shared tsc file-count run for the dead-export guard
  • smaller chain fixtures.

Production behavior is unchanged; every change is a test-only injection or a smaller fixture. Pass/fail counts: 8334 pass, 4 skip, 0 fail before; 8336 pass, 4 skip, 0 fail after (two new injection tests added).

Metrics

Measured on this machine, seeded one-process suite:
bun test ./src ./e2e ./evals ./scripts ./testkit --randomize --seed 424242.

Commit File Baseline After Saved
faea4d4c inject short teardown deadline src/subagent/run-shell-child-reap.test.ts 30.09s ~1.5s ~28s
70ec1fec inject scan ceiling src/plugins/read-file-guard-plugin.test.ts 5.71s 1.15s 4.5s
d5d460e8 share tsc file-count run scripts/check-dead-exports.test.ts 9.31s 7.15s ~2.0s
742e0435 shrink chain fixtures src/plugins/read-file-guard-plugin.test.ts 1.27s 0.40s ~0.87s
All four commits full suite 85.76s ~44.9s ~41s (~48%)

Notes:

  • Per-file savings do not sum to the suite total: 70ec1fec and 742e0435
    both change read-file-guard-plugin.test.ts (net 5.71s → 0.40s), and
    suite wall clock varies by a few seconds under machine load.
  • scripts/check-dead-exports.test.ts (ts-prune/tsc subprocess) is the main
    source of run-to-run variance, not these changes.
  • Pre-existing flake, not introduced here: one full-suite run under load
    showed 2 failures and 93.7s; every other run passes clean. Matches the
    known flaky 15s timeout in src/subagent/run-skill-scope.test.ts already
    logged in downloads/README.md.

Commits

faea4d4c inject short teardown deadline in wedged-close test

runSubAgent's end-of-turn teardown on the non-persist path waited out the
production 30s DEFAULT_CLOSE_DEADLINE_MS bound when agent.close() was
wedged by a live shell descendant. The wedged-close test in
src/subagent/run-shell-child-reap.test.ts triggers that path, so it paid
the full 30s wall clock every run.

Added an optional teardownDeadlineMs param to RunSubAgentParams that
overrides that bound, following the existing test-injection pattern
(outerRetryDelayMs, retryPolicy). Production never sets it and keeps the
30s default, so runtime behavior is unchanged. The test passes a 1s deadline
and still exercises the same contract: the bounded teardown fires on the
wedged close and settles the run.

Regenerated scripts/ci-timings.json per the documented policy (slow file
changed) so CI shard balancing uses the new per-file durations.

70ec1fec inject scan ceiling to shrink fixture

The single-line pagination test in src/plugins/read-file-guard-plugin.test.ts
built an 8MB fixture (READ_FILE_MAX_SCAN_BYTES + 4096) and paged through it
with repeated readFileBounded calls. Paging is stateless and line-offset
based: every hop opens a fresh stream at the start of the file and re-reads
all bytes up to its offset, so an 8MB one-line fixture cost ~168 hops, each
re-reading up to 8MB (~4s of the 5.7s file; quadratic in fixture size).

Added an optional maxScanBytes to ReadFileGuardPluginOptions that
overrides the scan ceiling, following the existing test-injection pattern
(whichExtractor, canExecuteHostCommands, teardownDeadlineMs).
Production never sets it and keeps the 8MB READ_FILE_MAX_SCAN_BYTES
default, so runtime behavior is unchanged. The scan-limit messages now
report the effective ceiling instead of the constant. The test injects a
256KB ceiling and keeps its fixture at ceiling+4096: same contract (single
line past the scan ceiling windowed across hops with exact reassembly, no
silent truncation, no scan-limit footer), ~6 hops over a 260KB file.

Regenerated scripts/ci-timings.json per the documented policy (slow file
changed) so CI shard balancing uses the new per-file durations.

d5d460e8 share tsc file-count run and inject it in the probe

scripts/check-dead-exports.test.ts was the largest file in the suite
(~9.3s, ~18%). Its cost came from running tsc --listFilesOnly three times
for the same deterministic count: once per floor test (~1.45s each) and a
third time inside the guard subprocess spawned by the end-to-end probe
(ts-prune ~4.6s + tsc ~1.5s).

  • The two floor tests now share one tsc run through a lazy module-level
    cache (liveScannedFileCount); each test still computes it on first use,
    so no test depends on another having run.
  • The guard (scripts/check-dead-exports.ts) accepts the count via the
    DEAD_EXPORT_GUARD_SCANNED_FILES env var (test injection only, same
    pattern as teardownDeadlineMs/maxScanBytes; production never sets it,
    so the gate always computes the count in normal runs). The end-to-end
    probe passes the cached real count, so the subprocess pays only ts-prune
    while still exercising the full gate wiring: config load, allowlist
    validation, real ts-prune over the real program, floor check, and exit 1
    naming the probe.
  • Added unit tests for the new injection parser (no override / positive
    integer pass, bogus values fail closed).

Regenerated scripts/ci-timings.json per the documented policy (slow file
changed) so CI shard balancing uses the new per-file durations.

742e0435 shrink chain fixtures via scan-ceiling injection

The two large-file pagination chain tests in
src/plugins/read-file-guard-plugin.test.ts paged a 45k-line (~11MB)
fixture through 200-line offset hops. Paging is line-offset based and
stateless: every hop opens a fresh stream at the start of the file and
re-reads all bytes up to its offset, so the cost is quadratic in fixture
size (225 hops x up to 11MB each, ~0.85s of the file's 1.27s).

The paging contract is ceiling-relative, so the chain tests now inject the
256KB scan ceiling (same value the single-line windowing test uses) and
keep the fixture above it: 4k lines (~1MB) instead of 45k. Same contracts
hold: no scan-limit trip on any hop, footer-driven continuation to the last
line, exact row reassembly, and a deep page (line 3500 of 4000) reachable
past the ceiling. The deep-page test passes the injection too, so its
"larger than the scan ceiling" property holds relative to the effective
ceiling. Production code is unchanged; the injection point already existed
from the prior commit.

Regenerated scripts/ci-timings.json per the documented policy (slow file
changed) so CI shard balancing uses the new per-file durations. Four-shard
balance is unchanged (~11.3s per shard).

The two floor tests each spawned tsc --listFilesOnly for the same
deterministic count, and the end-to-end probe paid for it a third time
inside the guard subprocess. Cache the count across the floor tests and
let the guard accept it via DEAD_EXPORT_GUARD_SCANNED_FILES (test
injection only; production never sets it), so the probe run pays only
ts-prune.
The two large-file pagination chain tests paged a 45k-line (~11MB)
fixture through 200-line offset hops. Paging is line-offset based:
every hop re-reads the file from the start, so the cost is quadratic
in fixture size (225 hops x up to 11MB each).

The paging contract is ceiling-relative, so the tests now inject the
256KB scan ceiling (same as the single-line windowing test) and keep
the fixture above it: 4k lines (~1MB) instead of 45k. Same contracts
hold: no scan-limit trip on any hop, footer-driven continuation to the
last line, exact row reassembly, and a deep page reachable past the
ceiling. Production code is unchanged.

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review · Approve

Shrinks the test-suite wall clock (~86s → ~45s) via test-injection seams (teardownDeadlineMs, maxScanBytes, DEAD_EXPORT_GUARD_SCANNED_FILES), smaller fixtures, and regenerated scripts/ci-timings.json; production behavior unchanged.

Findings

No findings.

Verification: diff stat is clean (no out-of-scope files, no Bin markers; touched prod files are exactly the three injection seams + the scan-limit message string), and all 7 commit subjects are typed test(...)/test: and ≤72 chars. Reviewer ran the targeted suites in the worktree: bun test src/plugins/read-file-guard-plugin.test.ts (41 pass — windowing/deep-page/contract checks intact), src/subagent/run-shell-child-reap.test.ts (5 pass — wedged-close settles in ~400ms via the 1s deadline), scripts/check-dead-exports.test.ts (32 pass, incl. "fails closed on a bogus value"), refresh-lock/session-refresh-race/test-parallel (16 pass), plus --randomize --seed 424242 runs (46 + 32 pass). Each seam defaults to the production constant when unset; DEAD_EXPORT_GUARD_SCANNED_FILES parsing rejects non-positive/non-integer values. ci-timings.json values track the measured per-file wall clocks with no contradiction.

@Gustav-Simonsson
Gustav-Simonsson merged commit 2365070 into corbitsdev:main Oct 7, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants