Repository navigation
ci: split heavy suites into parallel parts - #60
Merged
Merged
Conversation
Run the motd, Cloudflare Tunnel and push worker registration suites as four independent parts each, sharing a definitions lib, and execute a shard's files with a bounded worker pool (TEST_JOBS, serial by default). Serialize no behaviour change: the parts are contiguous slices of the original files that each parse standalone, so every assertion still runs. Also map modules/system-optimize.sh to its suite, lint tests/lib, fetch only one commit for the lint job, and run Debian's whole-tree bash -n once instead of per shard.
motd is the CI critical path (335 s of a 349 s run) and dominates its shard, so sharding by suite cannot shorten it. Give its top-level cases segment guards: each case is wrapped in 'if seg_skip; then :; else ... fi', and the guard runs everything unless TEST_SEGMENT is set. Only units whose text parses on its own are wrapped, so a guard never lands inside a subshell, loop, case or heredoc; function definitions, fixture writers and loose setup stay unguarded and therefore run in every process, which keeps shared fixtures available to every segment. TEST_SEGMENT is unset by default, so bash tests/test-motd.sh is unchanged. tests/run.sh learns TEST_JOBS (bounded parallelism over a shard's suites, serial and byte-identical output by default) and TEST_SEGMENTS (one process per segment for the suites listed as segmented). CI uses 4 for both. Also map modules/system-optimize.sh to its suite instead of falling back to the full run, fetch one commit for the lint job, and run Debian's whole-tree bash -n once instead of per shard. This reverts the per-file split of the three heavy suites: their parts depended on fixtures created by other parts, which the full run caught.
test-shards, test-selection and test-selection-git re-enter tests/run.sh with stub suites and assert the exact serial output, so they must not inherit TEST_JOBS or TEST_SEGMENTS from the CI job. run.sh also scrubs both from every suite it starts, so a nested invocation can never re-enable them.
Four segments re-ran the ~212 s of top-level code that cannot be wrapped (compound statements that do not parse standalone), so the shard went from 341 s to 390 s on Ubuntu and 317 s vs 306 s on Debian. Revert to the plain suite; the parallel runner and the other CI changes stay.
Each heavy suite keeps its original entry point and is rewritten into a shared
prologue plus four contiguous semantic segments guarded by 'if seg_is N; then'.
The prologue holds the definitions and the child-script fixtures that later
segments consume, so every segment runs on its own; the Cloudflare segments
1-3 additionally replay the parent-shell state (new_case, the two privilege
stubs, the uninstall transaction hook) that C0 used to establish for them.
Ranges follow the measured cost of the original linear run, so the largest
segment is about a third of the suite instead of the whole of it. Loops are
not split and no assertion, timeout or stress iteration is changed.
Verified on hkhkt (Debian 13, 8 vCPU), serial original vs four segments:
motd 655 PASS == 119+140+202+194, 473 s -> 162 s
push-worker 303 PASS == 90+70+85+58, 247 s -> 85 s
cloudflare not comparable there: the original suite already fails on
that host at 'uninstall INT at before-snapshot-create
returned 0', so equivalence is left to CI.
TEST_SEGMENTS now defaults to 1 and CI sets 4; with it unset every segment
runs in order, so a local 'bash tests/test-x.sh' still runs the whole suite.
TEST_JOBS=4 ran the four push-worker segments and test-push.sh at the same time, five processes on a four-core runner; test-push.sh then failed its transfer-failure assertion under contention. Segments already fill the runner, so run the suites of a shard one after another again (TEST_JOBS stays available, default 1).
They were expanded into consecutive entries of the job list, so with the pool off they ran one after another: the motd shard took 68+110+86+126 s, worse than the unsplit suite. Keep the segments of one suite in flight together and run whole suites of a shard only after that.
The suite re-executes itself through sudo when it is not root, and sudo resets the environment, so every segment saw an empty TEST_SEGMENT and ran the whole suite: four segments of 318 s each instead of a quarter each. Preserve the variable alongside PATH.
Segment boundaries now come from per-unit wall times measured on the shell test host, and may not fall inside the span of a function or top-level variable defined after the prelude: a segment only sees the prelude and its own block, so a cut inside such a span broke later segments.
motd, push-worker, push and cloudflare each get their own matrix shard, so cloudflare no longer queues behind xanmod inside other and the push worker suite no longer waits for test-push. push now covers test-push.sh only; tests/run.sh all remains the local full run.
Both suites were cut into four segments and run by a single job each, so their wall clock was the suite's total work divided by four cores. Cut each of them into eight segments (same single-file `seg_is` guards, prelude kept once, no overlap or gap) and add `TEST_SEGMENT_SET` to tests/run.sh so the two new shards of a suite run four segments each: motd-a / motd-b TEST_SEGMENTS=8, sets "0 1 2 3" and "4 5 6 7" cloudflare-a / -b same, two jobs Cut points come from per-statement wall times measured with a DEBUG trap on the shell test host (top-level statements and whole `for`/subshell blocks are atomic), minimising the largest segment; the reference tree in the same measurement passes 655 motd and 312 cloudflare assertions, exactly the per-segment sums after cutting. `TEST_SEGMENT_SET` accepts only distinct in-range indices and exits 2 before any suite starts; unset keeps the old behaviour. test-shards.sh now also proves that each source segment is covered by exactly one job, that the four subsets are the full segment set, and that invalid sets abort, and the README documents the seven shards.
L60-01: accept TEST_SEGMENTS from 1..8 only and TEST_SEGMENT_SET entries as a single digit 0..7, split with `read -a` so no value is subject to pathname expansion or long-integer arithmetic. Out-of-range, duplicate, oversized and glob-like entries exit 2 before the first suite starts. L60-02: parse every matrix entry as job/shard/suite/segments/segment_set without deduplicating, assert unique shards plus exactly-once full coverage separately for ubuntu-tests and debian-tests, bind each declared count to the suite's segment guards and to the family list in run.sh, and exercise subsets for all three segmented families. L60-03: check the whole execution plan against the per-suite segment counts (motd 8, cloudflare 8, push-worker 4) before running anything, keep TEST_SEGMENTS=1 as a serial whole-suite run, and refuse a subset aimed at an unsegmented suite. L60-04: drop TEST_SEGMENT and TEST_SEGMENT_SET from nested fixture environments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The CI critical path is three single suites: motd 335 s, Cloudflare Tunnel 258 s, push worker registration 217 s (run 40006297041, 349 s wall). Sharding by suite alone cannot go below the slowest file, so each of the three is split into four independent parts.
tests/test-{motd,cloudflare-tunnel,push-worker-registration}-NN.share contiguous slices of the original file;tests/lib/<family>-common.shholds the definitions and per-run setup that every part needs.bash -nsucceeds on both sides, so no part starts or ends inside a subshell, loop, case or heredoc, and the concatenation of the parts is the original file. Part 1 additionally runs the two/one prefix assertions that moved out of the lib, so each assertion executes exactly once.tests/run.shmaps every part to its family for PR selection and shard assignment, and runs a shard's files with a bounded worker pool:TEST_JOBS, serial by default (output identical to before), 4 in CI.modules/system-optimize.shnow maps totest-system-optimize.shinstead of falling back to full.tests/libis linted at error severity; the lint job fetches one commit; Debian runs the whole-treebash -nonce instead of per shard.Scope
Only CI wiring,
tests/run.sh, the three split families and README. No production script changes, no assertion removed or relaxed, no severity lowered.Validation
Local (iSH, aarch64 musl):
bash -nover every.sh;shellcheck -x --severity=warningfor production,--severity=errorfortests/andtests/lib/(rc=0);test-shards.sh,test-selection.sh,test-selection-git.sh,test-lint-coverage.sh,test-hup-parser.sh,test-ssh-security.shpass. Heavy suites cannot run here (no/dev/fd, per the repo's own rule that analysis and tests run in CI), so their behaviour is validated by this PR's full run.Expected impact
Per shard, four parts run concurrently on a 4 vCPU runner: motd ≈ 88 s, Cloudflare + xanmod ≈ 112 s, push worker + push ≈ 132 s, plus ~20 s of runner and dependency setup. Estimated wall 120–160 s versus 349 s measured. ShellCheck (95 s) is not parallelised because it is no longer on the critical path.
Risks
create_fixture,new_case,setup_fixture), so this is expected to hold; the full run in this PR is the check.TEST_JOBSoutside CI is 1: local behaviour is unchanged.Rollback
Revert the commit: the three original files are restored and
tests/run.shreturns to the serial loop. No branch protection or production configuration changes.