[roodboi]

Root cause

runtime-core test fixtures named their temporary roots <prefix>-<pid>-<clock> (or <prefix>-<pid> alone). On macOS SystemTime has microsecond resolution and crate::node::now() counts milliseconds, so two tests that cargo test starts together on parallel threads can compute the same name. Where the fixture then creates the root non-exclusively (state::private_directory is recursive), the second test silently shares the first one's root. In provider/hostname_authority/managed.rs this gives the hosted Candidate core (macos-latest) failure (run 39968507108): one test finds the other's provider lock held (provider_busy at :111), and the first fixture's Drop (remove_dir_all) deletes the root under the second (NotFound at :146, or the deadline assertion at :142). Where the create is exclusive, the same collision fails with File exists instead.

Fix (test-only; no product code path changes)

• New #[cfg(test)] module packages/runtime-core/src/test_root.rs, re-exported as crate::unique_test_root(parent, prefix) (mode 0777 less umask, the same as fs::create_dir) and crate::private_test_root(parent, prefix) (mode 0700). Each root is named <prefix>-<pid>-<sequence>. The sequence is a process-wide AtomicU64. The root is made with one exclusive, non-recursive mkdir. A name left by an earlier process with the same pid is skipped and never reused. Failures panic with a single value-free line (E_TEST_ROOT_CREATE / E_TEST_ROOT_EXHAUSTED).
• Lib fixtures that named roots from the pid alone or from pid plus clock now use the helper. Each fixture keeps its own parent (canonicalized or not), permissions, follow-up private_directory calls and cleanup: node/journal.rs (2), provider/bridge.rs, provider/config_audit.rs, provider/graph/normalized.rs, provider/graph/probes.rs (2), provider/graph/relay.rs, provider/image_load.rs (2), provider/network_tools.rs, provider/publication/recovery.rs, provider/source_sync.rs (2), provider/graph/host_relay/live_tests.rs (the control root, through a small ControlRoot::claim).
• Overlap with test(runtime): isolate fork-exposed host checks and make scratch names unique #399 (G40, HACK-1347). test(runtime): isolate fork-exposed host checks and make scratch names unique #399 also changes some of these fixtures. It appends crate::next_test_scratch() to the name and adds src/test_scratch.rs. For those fixtures this PR makes the byte-identical edit (same hunks, same test_scratch.rs, same lib.rs lines) instead of moving them to the new helper: hostname_authority/managed.rs (the failing test), project/snapshot.rs, config_audit.rs (recorded fixture), dependency_socket.rs, disk_audit.rs, disk_template.rs, engine.rs, guest_tool.rs, lifecycle.rs (5), lifecycle/interrupted.rs, lifecycle/prepared_boot/tests.rs, lifecycle/pressure_tests.rs, persistent_data/enrollment/witnessed/tests.rs, relay_loop/admission_tests.rs and source_transfer.rs. git merge-tree of this head with test(runtime): isolate fork-exposed host checks and make scratch names unique #399's head (cb6661fd) is clean, so the two PRs merge in either order.
• Fixtures the issue lists that already had a process-wide counter are unchanged: graph_cli/restore_tests.rs, project/adapter.rs, project/live_source.rs, project/registry.rs, project/snapshot/generated.rs and project/snapshot/mountpoints.rs.

Tests

• test_root::tests::parallel_fixtures_receive_distinct_freshly_created_roots: 32 threads behind one barrier each create 8 roots (mixed private and plain). The test asserts that all 256 are distinct, each is empty when returned, each holds only its own marker afterwards, and the private ones have no group or other bits.
• test_root::tests::a_root_left_by_an_earlier_process_is_skipped_and_never_reused: when the first three sequence names already exist, the fourth is returned, it is empty, and the stale roots are left untouched.
• Mutation proof:
  • Removing the counter (every name repeats) fails the parallel test: other threads exhaust their attempts on the one existing name with E_TEST_ROOT_EXHAUSTED.
  • Making AlreadyExists return the existing path (reuse) fails the stale test: it gets fixture-0 where fixture-3 was expected.

Evidence

• Repro (macOS, M-series, --test-threads=2 hostname_authority::managed, 200 runs each):
  • origin/next lib test binary: 30/200 and 41/200 failing runs, with the CI signatures (provider_busy at managed.rs:111; :142 / :146).
  • This branch: 0/200.
• Gates, all with cargo +1.97.1:
  • bun run check:local (fmt and clippy, all targets, -D warnings): pass.
  • All-feature clippy (--all-targets --all-features -D warnings): pass.
  • bun run test:local: lib 1050 passed / 60 ignored; bin 43 passed; all integration test binaries pass.
• Root bun run check (includes privacy-check) and bun run --cwd packages/cli typecheck: pass.
• gitleaks 8.30.1 over origin/next..HEAD: no leaks.
• Hosted CI at head a87aee14: every check passes, including Candidate core (macos-latest), Candidate core (ubuntu-latest), test, all four docker-e2e shards and all four Config compiler jobs.

Gaps

• The integration tests in packages/runtime-core/tests/ are separate crates and cannot use the #[cfg(test)] lib helper. Their remaining pid-only or pid-plus-clock names (stream_relay_contract.rs, relay_fence_contract.rs, quiescent_dependency_recovery_cli_contract.rs, source_device_rebind_cli_contract.rs, absent_recovery_cli_contract.rs, loopback_descriptor_churn.rs, cache_quarantine_cli_contract.rs) are each used by exactly one test, so they cannot collide inside one process. The only remaining risk is a stale name from an earlier process, which fails loudly.
• The src/main.rs unit tests (source_capture_tests.rs) are also outside the lib crate. They use two distinct prefixes and an exclusive create.
• Manual #[ignore] VM tests that build owner tokens from pid plus clock (source_transfer.rs, graph_probe_test.rs, source_failure_test.rs, gateway_probe_test.rs) are unchanged.
• Pre-existing, unrelated flake. Found while running the gates: provider::relay_owner::publication::tests::dead_publication_recovery_requires_unchanged_receipt_and_socket fails intermittently with NotFound at publication/tests.rs:299 (2/30 runs on origin/next, 1/30 on this branch). Under full-suite load its two dead_witness_* siblings once failed with relay_owner_refused at :384 / :432. Its roots already use a counter, so this PR does not address it; it is reported separately.

Refs HACK-1393

🤖 Generated with Claude Code

[linear-code]

HACK-1393