Repository navigation
test(native): pin Compose startup readiness timeout, transient-flag and strict-default bounds - #378
Merged
Merged
Conversation
…nd strict-default bounds Add tests/native-compose-readiness-bounds.test.ts for the readiness bounds that #362 (HACK-1279) added to waitNativeComposeStartupOwned: - A readiness timeout never becomes success, even when the post-deadline settled check finds the member rejoined and ready. - Each disjunct of the transient flag (first-scan running member, recheck running member, recheck placeholder) alone runs the settled check, which refuses the unchanged shape; restarting-only shapes, including a first-scan placeholder, do not; the flag follows the last observation. - Reads that do not opt into transient membership stay strict: strict and saved refuse every restart-race shape, and settled refuses the running shapes while keeping the restarting admission from before #362. A virtual readiness clock (Bun.sleep advances Date.now past the deadline) fixes the number of in-loop observations, and the fake Docker serves one scripted scan per inspect, so each bound is deterministic. Refs HACK-1305 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
This change adds tests only. HACK-1305 asks for tests that pin three Compose startup readiness bounds that #362 (HACK-1279) added in
src/lib/native-compose-ownership.ts:The #362 tests did not pin these bounds. I ran 10 source mutants that break them against
tests/native-compose-ownership.test.tsandtests/native-process-policy-initial-trace.test.ts(77 tests). None of the 10 was reliably killed (table below). The three "kills" in that run were replay-driver timeouts, not assertions (see Findings).The tests exposed no product defect, so production code is unchanged.
Fix
New
tests/native-compose-readiness-bounds.test.ts(7 tests):retry(the only selected on-failure service) ismember,absent-running,absent-restarting,placeholder-runningorplaceholder-restarting.webis always a healthy ID member.Bun.sleep) returns at once. The Nth poll movesDate.nowpast the deadline. The clock only moves between queries, so each query's own probe deadline is unaffected. This fixes the number of in-loop observations, and the inspect count shows whether the settled query ran. A harness-control test checks this for 1 and 3 observations.What each test pins:
retryrejoined and ready: the result isnull, the settled query ran (8 inspects), andobservedis never called. The same holds for a case-1 running recheck placeholder. An owned but never-ready observation times out asnull, without a settled query, and the last unready observation is not returned.first.transient || second.transient || recheck placeholdersis tested alone: first-scan running absent, recheck running absent, and a recheck placeholder for arestartingcontainer. Each alone runs the settled check, which refuses the unchanged shape with the original predicate and exact inspect count. Restarting-only shapes (absent in both scans, or a first-scan restarting placeholder) time outnullafter exactly one query, with no settled query. The flag follows the last in-loop observation: transient then restarting runs no settled query, and restarting then transient runs one.transientmembership stay strict. For every race shape (case 2, a first-scan running placeholder, and case-1 running and restarting recheck placeholders),settled, strict and saved reads refuse with the original predicate (live-endpoint-membershipornetwork-member-id), while the startup default keeps the shape unready.settledstill admits therestartingshapes from before #362, which strict and saved refuse. An unselected service is never admitted.Mutations
All mutants are in
src/lib/native-compose-ownership.ts. The "existing" column is the 77 tests in the two #362 test files.first.transientsecond.transientrestartingobservation counts as transient||=) instead of following the last observationrestartingmemberrestartingfirst-scan placeholdertransientsettledtransientMembershipdefault istruetransientresult.restarting &&from the wait's latch* Under the M2, M5 and M6 runs, one HACK-1279 trace replay test failed with
Initial process-policy replay unavailableafter 7–20 s, at a load average of about 35–180. These mutants cannot change those replays: M2 only edits the wait, and M5 and M6 only change thetransientfield, whichobserveNativeComposeStartupOwnedignores. The failures are replay-driver timeouts (the driver's 5 s default exec budget), not kills.The three survivors are equivalent:
restartingServices.has(...)fails.requireLiveMemberalso requiresmode === "startup", andnetworkMembersrequiresopts.startup. Flipping the default alone cannot change any outcome. This is the same as M6 in fix(native): keep on-failure restart-backoff and recheck-placeholder members unready until the readiness deadline #362. The new tests pin the observable bound instead: every non-transient read refuses every race shape.result.transientimpliesresult.restarting. Every disjunct requires an admitted container: a transient-admitted running member is added torestarting, and a claimed recheck placeholder owner is never an ID member, sorequireLiveMemberadmits it too.Evidence
mise x bun@1.4.2, afterbun install --frozen-lockfile. No lockfile changes.bun test tests/native-compose-readiness-bounds.test.ts: 7 pass on this Mac, at a load average of up to about 180.bun run --cwd packages/cli typecheck: clean.bun run --cwd packages/cli check: 0 errors, 34 warnings, the same as the base.ultracite checkon the new file is clean. Roottscreports no diagnostics in the new file.gitleaks dir(8.30.1, repository config) over agit log -p origin/next..HEADexport: no leaks.f2aec257, run with other units' suites in parallel (load average about 100–160): 7548 pass, 126 skip, 33 fail, 7 errors in 541 files (70 min). The new file passed.--timeout 60000(load average about 11–26):env-key-argument-redaction-cli,native-authored-project-oneoff,native-authored-project-exec,native-compose-proxy-probe,native-project-branch-cli,registry-touch-worktrees,portable-bootstrap-env-refusal,prerelease-install,native-compose-warm-up-commandandproject-env-config. The newnative-compose-readiness-boundsfile also passes (7/7).native-compose-retained-storage-command(5/5),native-compose-storage-fence-cost(4/4),native-compose-adoption-generation(13/114, all at their own 20–30 s test timeouts),native-authored-process-incarnation(1/10) andnative-authored-project-recovery(2/18).ec223cdb(retained-storage 3/5, fence-cost 4/4). Theprocess-incarnationandno-hostrecovery failures also show in other units' M3 logs on base.test, all fourdocker-e2eshards,linux-process-lifetimeandsecret-scan.Gaps
src/lib/native-compose-command.tsis not pinned here. Anullreadiness givesvalue: code || 1andoutcome: "uncertain". Pinning it needs the native authored-project start harness, andtests/native-authored-project-start.test.tsalready fails on base.opts.ownership.signal?.abortedskips the settled query) is not pinned. Aborting deterministically between the last observation and the latch check needs a probe hook.Findings (not fixed here)
tests/e2e/native-process-policy-initial-replay.tsuses a 5 s defaulttimeoutMs. Under host load it fails withInitial process-policy replay unavailableon tests that the source change cannot affect (see the footnote above). Severity: low (flake).runningon-failure member" was missing or placeholdered. The code also runs it for arestartingcontainer whoseep-key first appears in the recheck scan, which then refuses withnetwork-member-idat the deadline. Thetransientfield's own doc ("an admitted container depended on a shape that asettledquery refuses") matches the code, and the new tests pin the code. Severity: low (doc precision).bun test --timeout 60000 ./tests/native-compose-retained-storage-command.test.ts. It fails atec223cdband atf2aec257. Expected: the fake engine's coldup --detach --jsonexits 0. Actual: it exits 1 after about 34 s withE_CONFIG_INVALID,native_compose_effect_refusal: { stage: "storage-helper-start", reason: "unclassified" }.native-compose-storage-fence-cost.test.tsfails 4/4 the same way. Hosted Linux CI passes. This is possibly a known flake. Severity: medium, because it hides the storage regression tests on macOS hosts.Refs HACK-1305
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.