Skip to content

test(runtime-core): size release-check scans for load and pin the drop, kill-anchor and retired-format guards - #501

Draft
roodboi wants to merge 4 commits into
nextfrom
wf/g107-runtime-core-load-and-guards
Draft

roodboi wants to merge 4 commits into
nextfrom
wf/g107-runtime-core-load-and-guards

Conversation

@roodboi

@roodboi roodboi commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Follow-ups from the #484, #485 and #486 verifiers. This changes test code, with one exception: Reservation::drop is refactored behind a named budget, and its behavior is unchanged. Refs HACK-1457

Root cause and fix

1. Load-sensitive release-check tests

Cause. Every failing test reaches lifecycle::lock_absent_disks, which runs a whole-host lsof over the owned disks with the production STOP_HANDLE_WAIT of 5 s. lsof walks every process on the host. Under load, and with other tests' isolated children scanning at the same time, one scan can take longer than 5 s. The fixture then fails as provider_timeout_uncertain before it reaches its result under test.

I reproduced this on next (246f1b4) by adding 20 CPU spinners and 20 /bin/sh -c true spawn loops (load ~105-120). Run 3 of 3 failed 7 tests, all with provider_timeout_uncertain or the isolated parent of such a child:

  • interrupted::unadopted_unclean_stop_child (interrupted.rs:300, the one in the issue)
  • admission_pool::uncertain_stop_rewrite_child
  • host_filesystem::device_migration_child and substituted_lock_child
  • short_home::explicit_recovery_child, vm_lock_release_child and active_socket_retry_child

The G105 verifier logs show the same set at load 78-137.

Fix. The unit-test build gets a named budget instead of the fixed 5 s. STOP_HANDLE_WAIT is test_executable::LOADED_HOST_SCAN_WAIT (60 s) under cfg(test). The production value (5 s) is unchanged. These fixtures prove what the scan reports (status 1 and no output), not how fast it reports it, so what they prove is unchanged. Tests of the timeout path itself already pass their own bound to lock_absent_disks_within. 60 s is still well inside ISOLATED_HANG_GUARD (120 s), so a scan that hangs is still reported.

The loaded runs found one more fixed bound, a 2 s wait for a spawned /bin/sh in private_child tests. It is a hang guard, so it is now FIXTURE_SHELL_HANG_GUARD (60 s), with a closed E_TEST_FIXTURE_SHELL_HANG message.

I also tightened unadopted_unclean_stop_child's second check. It used to accept any error (is_err()), so a scan timeout also satisfied it. It now requires the identification error itself (provider_state) and requires that no disk was recorded.

2. Reservation::drop's unnamed 5 s

The wait is now RELEASE_LEASE_WAIT, and the drop logic moved into Reservation::release_within(wait) -> Release. The fallback is documented. If the lease stays busy, or the lease, home or receipt does not verify, the receipt is left as written and its slots stay reserved. It is then released by recover_orphan (never-admitted runs) or by dead-owner graph cleanup (recover_cleaned). Both require the recorded owner to be dead. The behavior is unchanged.

New test a_release_that_outlasts_a_busy_lease_leaves_a_recoverable_receipt checks four things:

  • a free lease removes the receipt
  • a held lease returns LeaseBusy only after the wait has elapsed
  • the receipt bytes and its reserved slot are unchanged after that
  • recover_record, the shared core of both recovery paths, then releases the receipt under the lease, and a late retry does not touch the receipt again

3. Claim-writer failure attribution

Cause. A writer that panicked inside the lease never decremented the shared "inside" counter. Every later writer then failed with E_TEST_CLAIM_LEASE_OVERLAP, and join().unwrap() hid the original message.

Fix. An InsideLease drop guard is declared after the lease guard, so it drops first and leaves the count before the lease is released. Joins now collect every writer's panic message in writer order (E_TEST_CLAIM_WRITER_PANICKED (writer N: …)). New test a_writer_panic_inside_the_lease_names_itself_not_an_overlap injects a panic in writer 0 and requires a failure that names only that panic.

4. #486 gaps (src/node/)

  • Kill anchor. New test repeated_kills_of_a_child_that_never_exits_keep_the_first_kill_anchor. The child is not a group leader, but its recorded identity says it is, so every group signal is authorized and finds no group (ESRCH, tolerated). The child outlives each SIGKILL and is re-signalled on every later pass. The test requires Child termination deadline exceeded., at least three signal checks, and the child still alive afterwards. A named 60 s hang guard turns a regression into a failure instead of a hang.
  • Retired-format fixture. The upgraded-node fixture is now the live child's own identity (observe(pid)) with only boot in the retired seconds:micros form. The old fixture also had start: 1. Note that schedule never calls verify: the "never signals" result comes from the structure. So the test now checks the only signal authority directly. verify(legacy, live) and verify(legacy, legacy) must both refuse with the retired-format message.

5. IPv6 probe branch

New test peer probe fetches an IPv6-only answer through a bracketed host. The stubbed Bun.dns.lookup returns only {address: "::1", family: 6}, and the listener is on ::1. The test requires exit 0 and that both requests arrive with Host: [::1]:port. runStubbedPeerProbe gained an optional answer for this.

Mutation proofs (each reverted)

# Mutation Result
M1 Base binary (5 s scan budget) under the same churn load 1 of 3 full runs failed, 7 tests, all provider_timeout_uncertain
M2a Remove the receipt when the lease wait expires E_TEST_RECEIPT_CHANGED
M2b Stop waiting for the lease (single attempt) E_TEST_RELEASE_WAIT_NOT_HONORED
M2c Never remove the receipt under a free lease E_TEST_RELEASE_NOT_REMOVED
M3 InsideLease skips its decrement while panicking (old behavior) E_TEST_CLAIM_FAILURE_MISATTRIBUTED; the message lists writers 1-7 as E_TEST_CLAIM_LEASE_OVERLAP
M4a killed.get_or_insert_with(Instant::now) changed to killed = Some(Instant::now()) E_TEST_KILL_GRACE_REANCHORED after the 60 s guard; the 7 existing supervisor tests still pass
M4b Remove the retired-format guard in ownership::verify E_TEST_RETIRED_FORMAT_NOT_REFUSED
M5 target.hostname=x.address (no brackets) IPv6 test fails, exit 26 (rejected)

Evidence

  • Under load. I ran full lib runs (1,068 tests) with 20 CPU spinners and 20 spawn-churn loops, plus whatever else was running on the host. A trap stops the generator, and none was left running.
    • Batch 1 (commit a3b87e8, load 101-242): 29 of 30 passed. The one failure, at load 228, was a different fixed bound: private_child::independently_absent_guest_allows_reaping_failed_or_live_outer_transport waited 2 s for a spawned /bin/sh to exit. Commit 8d591db turns that wait and its sibling into FIXTURE_SHELL_HANG_GUARD (60 s).
    • Batch 2 (final head 8d591db, load 75-249): 29 of 30 passed. The one failure was https_recovery::full_legacy_command_child, which got https_recovery_refused after 1.3 s. That is not a wall-clock bound; see Gaps.
    • Across all 60 runs, no release-check test failed and no provider_timeout_uncertain appeared. The base commit under the same generator had failed 7 such tests in 1 of 3 runs.
  • Focused. private_child tests pass: 8. dependency_slots::tests, node:: and interrupted tests pass: 75 passed. tests/native-networks-fixture.test.ts passes: 67 tests.
  • Gates.
    • bun run check:local (fmt and clippy -D warnings, cargo 1.97.1) passes.
    • All-feature clippy (--all-targets --all-features, with Zig 0.15.2) passes.
    • Root bun run check (with privacy:check) and bun run --cwd packages/cli typecheck pass.
    • gitleaks 8.30.1 over origin/next..HEAD finds no leaks.
  • Full suite. bun run test:local (cargo 1.97.1, no added load) at 8d591db: all 21 test binaries passed, including the lib (1,068 passed). Hosted CI at 8d591db is all green: test shards 1-4, Candidate core (ubuntu and macos), docker-e2e shards 1-4, Config compiler on 4 platforms, Runtime state models, linux-process-lifetime, runtime-images and secret-scan.

Gaps

  • https_recovery::full_legacy_command_child failed once in 30 loaded runs, with https_recovery_refused after 1.3 s. The fixture binds an ephemeral port, drops it, and later requires port_absent on that port, so another socket on the host can take the port in between. This is a port-reservation race, not a timing bound, so it is reported as a finding and not changed here. The refusal is closed (no stage is reported), so I have not confirmed which check refused.
  • Other short fixed deadlines remain in the lib tests (host_endpoint, relay_loop and https_recovery tests use 1-3 s waits). None of them failed in the 60 loaded runs, so they were left as they are.
  • The production STOP_HANDLE_WAIT (5 s) is unchanged. admission_pool.rs documents lsof taking 5.4-6.5 s at load 30-80, so down and recover on a loaded host can still fail as provider_timeout_uncertain. This is reported as a finding and not changed here, because it changes product behavior.
  • network_update::stopped still has a fixed, unnamed 5 s lsof budget. No lib test reaches it after the scan, so it is not load-sensitive in the suite. It is left as is.
  • The kill-anchor test models a SIGKILL that never takes effect with a non-leader child and an injected leader identity. It does not use a real unkillable process.
  • The Docker e2e suite was not run locally, per policy.

🤖 Generated with Claude Code

…p, kill-anchor and retired-format guards

- Unit-test builds give the stop paths' whole-host lsof scan a named
  LOADED_HOST_SCAN_WAIT (60 s) instead of the 5 s production budget, which
  expired under load and failed release-check fixtures as
  provider_timeout_uncertain before their result under test.
- Name Reservation::drop's lease wait (RELEASE_LEASE_WAIT), document the
  dead-owner recovery fallback, and prove a receipt left by an expired wait
  is unchanged and recoverable.
- A claim writer that panics inside the lease is reported as that panic,
  not as a lease overlap by the writers after it.
- Pin the once-only SIGKILL grace anchor and make the upgraded-node fixture
  differ from its live child only in boot format.
- Cover the IPv6 branch of the peer probe's address rewrite.

Refs HACK-1457

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 24, 2026

Copy link
Copy Markdown

HACK-1457

roodboi and others added 3 commits October 24, 2026 18:31
…hang guard

A 2 s wait for a spawned /bin/sh to exit or print its marker expired at load
average ~230 during the HACK-1457 load runs. It is a hang guard, not a
correctness bound, so it becomes FIXTURE_SHELL_HANG_GUARD (60 s) with a
closed E_TEST_FIXTURE_SHELL_HANG message.

Refs HACK-1457

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds and pick unallocated ports

The second round of HACK-1457 load fixes. The CPU counter test now spins for
40 ms of thread CPU and brackets the native read between an inner and an outer
process-clock window, so parallel CPU use cannot fail it. HTTPS recovery and
discovery fixtures take a port below the kernel's ephemeral range, which no
port-0 bind or outgoing connection can take before the fixture needs it.
host_endpoint, relay_loop and https_recovery tests replace their fixed 1-6 s
waits with LOADED_EVENT_HANG_GUARD, MAX_CONNECT_BUDGET or ISOLATED_HANG_GUARD;
expiry tests still force an expired deadline. A new test covers the
Reservation Drop wiring.

Refs HACK-1457

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant