[roodboi]

Summary

Fixes the three intermittent Bun test failures in HACK-1422. One of them is a real product bug. It is not the lost update the ticket suspected: #435's layer lock already covers the legacy dotenv path. It is a false refusal: a concurrent hack env set/unset on a .hack/.env* file can exit 1 with the misleading "symlink or not a regular file" message. No data is lost.

1. Legacy dotenv concurrency (tests/project-env-layer-lock.test.ts): product fix

Root cause. upsertDotEnvValue and removeDotEnvKey read the env file once without the layer lock as a preflight. The preflight refuses a symlinked file before the lock creates .internal/, and remove uses it to skip the lock on a no-op. Every env resolver also reads without the lock. readReplaceablePrivateFile does lstat, then a no-follow open, and refuses if the two inodes differ. A lock-holding writer's atomic rename that lands between those two calls therefore failed the reader with PRIVATE_ENV_FILE_REFUSAL, although the path only ever held regular files. The failing stacks were all readPrivateEnvFile <- upsertDotEnvValue:526 / removeDotEnvKey:556, which are the two unlocked preflights. On base under load the whole file failed 3 of 5 runs, each time with that refusal.

Fix (src/lib/private-file.ts). When the lstat entry and the opened entry are both regular files but differ, the read starts again, at most REPLACEABLE_READ_ATTEMPTS (8) times. That returns the previous or the new contents, which is what replacePrivateFileAtomic already promises readers. The other refusals behave as before:

• A symlink (ELOOP) or non-regular entry is still refused at once, on any attempt.
• A path replaced on every attempt is still refused closed.
• The directory identity recheck is unchanged.

Tests (tests/private-env-file.test.ts). These intercept open so the replacement lands exactly between lstat and open. They are deterministic and need no load.

• A set whose unlocked read races another writer's rename succeeds and keeps both keys.
• Negative control: a file replaced on every attempt is refused closed after exactly REPLACEABLE_READ_ATTEMPTS opens, and no lock directory is created.
• Negative control: a symlink swapped in during that window is still refused after one open, and the outside file is untouched.

Mutation proof.

• Restore the old refusal on an inode mismatch: the race test fails (exit 1 with the refusal), and so does the bounded-retry control (expected 8 opens, received 1).
• Widen the retry bound ×100: the bounded-retry control fails (expected 8, received 800).

2. tests/native-runtime-client.test.ts, "authored * finishes while a descendant retains stdout and stderr": test fix

Root cause. The completion wait used waitForFile, which only waits for the file to exist. The keeper's writeFileSync creates the file before it writes finished, so the final toBe("finished") could read "".

Fix. A new waitForFileText(path, expected, budget = HANDSHAKE_BUDGET_MS) waits, with a bound, for the exact text. The assertion is unchanged.

Mutation proof. The keeper now always opens the empty-file window: it writes '', sleeps TORN_HANDSHAKE_GAP_MS, then writes finished, the same pattern as the file's existing pid handshake. With the old existence wait, all 6 cases fail with Expected "finished", Received "". With the content wait, all pass.

3. tests/native-compose-storage-witness-xattr-owner.test.ts, "same-birth root replacement while awaiting the final reader…": named budget

Cause: Bun's 5 s default timeout under load, not an unbounded wait. The case runs entirely in process against a fake carrier. The carrier deadline is 60 s, and no timer or poll is involved. On this host (load average about 40 to 50 on 18 cores) the case alone took 7.3 s wall time with 5.3 s of CPU, 4 s of it system time from the durable store's file work. With the default timeout it failed in both runs here (standalone and whole file) at about 5002 ms. A whole-file run also timed out 4 other in-process store cases at 5 s.

Fix. A named STORE_CASE_BUDGET_MS = 30_000, documented with this reason, applied to the named case and to the 3 other test sites observed timing out (command storage carriers use the %s named storage budget…, post-workload marker loss…, unknown enrolled helper %s blocks replay…). Biome reformatted those 4 cases into the multi-line test(name, fn, budget) shape, so most of that file's diff is indentation. git diff -w shows +34/−9.

Evidence

All local runs used a private HOME, HACK_HOME and DOCKER_CONFIG, with HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK=1, on Bun 1.4.2 (pinned). Each loop ran 30 times in a row with 8 extra busy-loop processes, stopped by an EXIT/INT/TERM/HUP trap, on a host already at load average about 40 to 50 on 18 CPUs.

Target	Result
xattr-owner same-birth root replacement… (-t)	30/30 passed (with Bun's 5 s default this case failed in both runs here, at about 5002 ms)
tests/private-env-file.test.ts (whole file, includes the new race tests)	30/30 passed
native-runtime-client finishes while a descendant retains… (6 cases)	30/30 passed (all 6 cases each run)
tests/project-env-layer-lock.test.ts (whole file, includes the legacy dotenv case)	30/30 passed (12/12 cases each run)

Before the fix, the same loop on project-env-layer-lock.test.ts failed 3 of 5 runs. Two earlier runtime loops were discarded and rerun in full on the committed tree: in one, I staged files mid-run, which tripped the checkout write guard while the 6 cases passed; in the other, the host stalled for about 7 h during a run, so the wall-clock drain assertion saw a 25264 s gap.

Gates at the PR head:

• gitleaks dir over a git log -p origin/next..HEAD export: no leaks.
• Root bun run check: passed, warnings only.
• bun run --cwd packages/cli typecheck: 0 errors.
• Related focused files passed: compose-env-at-rest*, project-env-config, project-env-write-refusals, secret-store*, shared-host-lifecycle. The exception is one hack-env.test.ts case that also fails on base when HACK_HOME is set (see Gaps).
• No CLI surface or consumer guidance changed.

The full-suite evidence is hosted CI at this exact head (8ff17289): test and all 4 test shards passed, and so did every other check (docker-e2e ×4, candidate core, config compiler, secret-scan and the others).

Gaps

• The xattr-owner file has many other in-process store cases still on the 5 s default. Only the named case and the 3 sites observed timing out got the budget. Others could time out on a loaded runner the same way.
• The product fix covers every readReplaceablePrivateFile caller: env files, project env layers, the secret store, compose env at rest and shared host lifecycle. Only the env-file path has a dedicated race test. The others share the same code path.
• tests/hack-env.test.ts "resolveHackEnv skips encrypted backend access…" fails whenever HACK_HOME is set. It writes its global config to $HOME/.hack and does not clear HACK_HOME/HACK_GLOBAL_CONFIG_PATH. This is pre-existing on base, CI does not set HACK_HOME, and it is outside this unit.

Round 2: xattr-owner budget no longer races the fake carrier deadline

Verifier defect. At 8ff17289 the same-birth case did not pass 30 runs in a row on a host at load average 45 to 95 on 18 CPUs (28/30, 13/30, 14/30, each failure a clean 30 s timeout). Raising the budget alone was not enough. The fake carrier's fixed Date.now() + 60_000 deadline then expired first, and the case failed with the storage witness refusal and exchanged false instead of E_NATIVE_COMPOSE_UNCERTAIN.

Why the case is slow. Its cost is a fixed amount of store file work, not a wait. With the calls counted through spies, the case's second up alone makes about 19k lstat, 13k realpath and 4.5k open calls, the same count on every run whatever the wall time. Most come from about 1.3k store check() calls, each rechecking about 7 held directories. Wall time therefore follows host load. A deep private TMPDIR makes each realpath more expensive, which helps explain why local runs under a scratch TMPDIR were slower than hosted CI.

Fix (tests/native-compose-storage-witness-xattr-owner.test.ts only):

• FAKE_CARRIER_LIFETIME_MS = 2 * STORE_CASE_BUDGET_MS replaces the fixed 60 s in fake(). Every carrier is captured inside a case, so its deadline is always later than that case's Bun budget. A slow host can now fail a case only as a clean timeout, never with a misleading refusal.
• STORE_CASE_BUDGET_MS goes from 30 s to 180 s. It is a hang guard, not a speed assertion: no case it covers waits on a timer.
• The cheaper-case option was not taken. The cost is in the product's store rechecks, and this unit does not change product code (see Findings).

Mutation proof.

• Carrier lifetime set to 5 s (shorter than the case): the case fails at line 2237 with expect(exchanged).toBe(true), received false. That is exactly the verifier's failure mode.
• Carrier lifetime set to 1.5 s: setup fails with E_NATIVE_COMPOSE_UNCERTAIN thrown from active().
• With the committed value, the case passes.

Evidence (Bun 1.4.2; private HOME, HACK_HOME, DOCKER_CONFIG and a deep scratch TMPDIR; HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK=1; 8 extra busy loops stopped by an EXIT/INT/TERM/HUP trap):

Target	Result
xattr-owner same-birth root replacement… (-t), 30 runs in a row	The case passed 30/30, at load average 53 to 245 on 18 CPUs. Its duration ranged from 5.6 s to 79.4 s (median about 24 s). One run (79.4 s) went past the old 60 s carrier deadline, and another took 59.8 s.

In run 12 of that loop the case passed, but the file run exited 1. The checkout write guard's afterAll hook (tests/helpers/checkout-write-guard.ts) timed out after 5 s running git status under load ("killed 1 dangling process"). That hook is outside this unit; see Findings.

Merge. next (1281184b) is merged in; the merge was clean.

Gates at the new head 34e6af28:

• gitleaks dir over a git log -p origin/next..HEAD export: no leaks.
• Root bun run check: passed (privacy-check ok, turbo 2/2).
• bun run --cwd packages/cli typecheck: 0 errors.
• No CLI surface or consumer guidance changed.
• Hosted CI at exact head 34e6af28 (run 40110572468, attempt 1): all 20 checks passed. That includes test and test shards 1 to 4 (the full-suite evidence), docker-e2e and shards 1 to 4, Candidate core ×2, Config compiler ×4, Runtime state models, linux-process-lifetime, runtime-images and secret-scan.

Remaining gap (unchanged from round 1, now measured). A local run of the whole xattr-owner file at the new head, at load average about 20 to 60, timed out 17 other cases. All 17 were timeouts, at Bun's 5 s default or at the file's existing 30_000 literals. The E_NATIVE_COMPOSE_STATE errors in that log are knock-on effects after those timeouts. Those cases do the same store work, so they share the same exposure to a loaded host. Hosted CI passes them.

Findings (not fixed here)

• Store recheck cost (product, low to medium). One up with one storage witness makes about 1.3k full store check() calls: about 19k lstat, 13k realpath and 4.5k open. That is several seconds of system time on a loaded host, and it grows with the depth of the project path. Expected: rechecks that do not grow with every proof step. Actual: about 7 held directories rechecked about 1.3k times per up.
• Checkout write guard hook timeout (test infrastructure, low). installCheckoutWriteGuard's afterAll runs a synchronous git status under Bun's default 5 s hook timeout. Under load it timed out once in 30 runs and failed a file whose cases all passed. Expected: the guard never fails a passing file for being slow. Actual: a beforeEach/afterEach hook timed out.

Refs HACK-1422

🤖 Generated with Claude Code

[linear-code]

HACK-1422