Repository navigation
Conversation
…p, kill-anchor and retired-format guards - Unit-test builds give the stop paths' whole-host lsof scan a named LOADED_HOST_SCAN_WAIT (60 s) instead of the 5 s production budget, which expired under load and failed release-check fixtures as provider_timeout_uncertain before their result under test. - Name Reservation::drop's lease wait (RELEASE_LEASE_WAIT), document the dead-owner recovery fallback, and prove a receipt left by an expired wait is unchanged and recoverable. - A claim writer that panics inside the lease is reported as that panic, not as a lease overlap by the writers after it. - Pin the once-only SIGKILL grace anchor and make the upgraded-node fixture differ from its live child only in boot format. - Cover the IPv6 branch of the peer probe's address rewrite. Refs HACK-1457 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hang guard A 2 s wait for a spawned /bin/sh to exit or print its marker expired at load average ~230 during the HACK-1457 load runs. It is a hang guard, not a correctness bound, so it becomes FIXTURE_SHELL_HANG_GUARD (60 s) with a closed E_TEST_FIXTURE_SHELL_HANG message. Refs HACK-1457 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ntime-core-load-and-guards-r2
…ds and pick unallocated ports The second round of HACK-1457 load fixes. The CPU counter test now spins for 40 ms of thread CPU and brackets the native read between an inner and an outer process-clock window, so parallel CPU use cannot fail it. HTTPS recovery and discovery fixtures take a port below the kernel's ephemeral range, which no port-0 bind or outgoing connection can take before the fixture needs it. host_endpoint, relay_loop and https_recovery tests replace their fixed 1-6 s waits with LOADED_EVENT_HANG_GUARD, MAX_CONNECT_BUDGET or ISOLATED_HANG_GUARD; expiry tests still force an expired deadline. A new test covers the Reservation Drop wiring. Refs HACK-1457 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups from the #484, #485 and #486 verifiers. This changes test code, with one exception:
Reservation::dropis refactored behind a named budget, and its behavior is unchanged. Refs HACK-1457Root cause and fix
1. Load-sensitive release-check tests
Cause. Every failing test reaches
lifecycle::lock_absent_disks, which runs a whole-hostlsofover the owned disks with the productionSTOP_HANDLE_WAITof 5 s.lsofwalks every process on the host. Under load, and with other tests' isolated children scanning at the same time, one scan can take longer than 5 s. The fixture then fails asprovider_timeout_uncertainbefore it reaches its result under test.I reproduced this on
next(246f1b4) by adding 20 CPU spinners and 20/bin/sh -c truespawn loops (load ~105-120). Run 3 of 3 failed 7 tests, all withprovider_timeout_uncertainor the isolated parent of such a child:interrupted::unadopted_unclean_stop_child(interrupted.rs:300, the one in the issue)admission_pool::uncertain_stop_rewrite_childhost_filesystem::device_migration_childandsubstituted_lock_childshort_home::explicit_recovery_child,vm_lock_release_childandactive_socket_retry_childThe G105 verifier logs show the same set at load 78-137.
Fix. The unit-test build gets a named budget instead of the fixed 5 s.
STOP_HANDLE_WAITistest_executable::LOADED_HOST_SCAN_WAIT(60 s) undercfg(test). The production value (5 s) is unchanged. These fixtures prove what the scan reports (status 1 and no output), not how fast it reports it, so what they prove is unchanged. Tests of the timeout path itself already pass their own bound tolock_absent_disks_within. 60 s is still well insideISOLATED_HANG_GUARD(120 s), so a scan that hangs is still reported.The loaded runs found one more fixed bound, a 2 s wait for a spawned
/bin/shinprivate_childtests. It is a hang guard, so it is nowFIXTURE_SHELL_HANG_GUARD(60 s), with a closedE_TEST_FIXTURE_SHELL_HANGmessage.I also tightened
unadopted_unclean_stop_child's second check. It used to accept any error (is_err()), so a scan timeout also satisfied it. It now requires the identification error itself (provider_state) and requires that no disk was recorded.2.
Reservation::drop's unnamed 5 sThe wait is now
RELEASE_LEASE_WAIT, and the drop logic moved intoReservation::release_within(wait) -> Release. The fallback is documented. If the lease stays busy, or the lease, home or receipt does not verify, the receipt is left as written and its slots stay reserved. It is then released byrecover_orphan(never-admitted runs) or by dead-owner graph cleanup (recover_cleaned). Both require the recorded owner to be dead. The behavior is unchanged.New test
a_release_that_outlasts_a_busy_lease_leaves_a_recoverable_receiptchecks four things:LeaseBusyonly after the wait has elapsedrecover_record, the shared core of both recovery paths, then releases the receipt under the lease, and a late retry does not touch the receipt again3. Claim-writer failure attribution
Cause. A writer that panicked inside the lease never decremented the shared "inside" counter. Every later writer then failed with
E_TEST_CLAIM_LEASE_OVERLAP, andjoin().unwrap()hid the original message.Fix. An
InsideLeasedrop guard is declared after the lease guard, so it drops first and leaves the count before the lease is released. Joins now collect every writer's panic message in writer order (E_TEST_CLAIM_WRITER_PANICKED (writer N: …)). New testa_writer_panic_inside_the_lease_names_itself_not_an_overlapinjects a panic in writer 0 and requires a failure that names only that panic.4. #486 gaps (
src/node/)repeated_kills_of_a_child_that_never_exits_keep_the_first_kill_anchor. The child is not a group leader, but its recorded identity says it is, so every group signal is authorized and finds no group (ESRCH, tolerated). The child outlives each SIGKILL and is re-signalled on every later pass. The test requiresChild termination deadline exceeded., at least three signal checks, and the child still alive afterwards. A named 60 s hang guard turns a regression into a failure instead of a hang.observe(pid)) with onlybootin the retiredseconds:microsform. The old fixture also hadstart: 1. Note thatschedulenever callsverify: the "never signals" result comes from the structure. So the test now checks the only signal authority directly.verify(legacy, live)andverify(legacy, legacy)must both refuse with the retired-format message.5. IPv6 probe branch
New test
peer probe fetches an IPv6-only answer through a bracketed host. The stubbedBun.dns.lookupreturns only{address: "::1", family: 6}, and the listener is on::1. The test requires exit 0 and that both requests arrive withHost: [::1]:port.runStubbedPeerProbegained an optionalanswerfor this.Mutation proofs (each reverted)
provider_timeout_uncertainE_TEST_RECEIPT_CHANGEDE_TEST_RELEASE_WAIT_NOT_HONOREDE_TEST_RELEASE_NOT_REMOVEDInsideLeaseskips its decrement while panicking (old behavior)E_TEST_CLAIM_FAILURE_MISATTRIBUTED; the message lists writers 1-7 asE_TEST_CLAIM_LEASE_OVERLAPkilled.get_or_insert_with(Instant::now)changed tokilled = Some(Instant::now())E_TEST_KILL_GRACE_REANCHOREDafter the 60 s guard; the 7 existing supervisor tests still passownership::verifyE_TEST_RETIRED_FORMAT_NOT_REFUSEDtarget.hostname=x.address(no brackets)rejected)Evidence
private_child::independently_absent_guest_allows_reaping_failed_or_live_outer_transportwaited 2 s for a spawned/bin/shto exit. Commit 8d591db turns that wait and its sibling intoFIXTURE_SHELL_HANG_GUARD(60 s).https_recovery::full_legacy_command_child, which gothttps_recovery_refusedafter 1.3 s. That is not a wall-clock bound; see Gaps.provider_timeout_uncertainappeared. The base commit under the same generator had failed 7 such tests in 1 of 3 runs.private_childtests pass: 8.dependency_slots::tests,node::andinterruptedtests pass: 75 passed.tests/native-networks-fixture.test.tspasses: 67 tests.bun run check:local(fmt and clippy-D warnings, cargo 1.97.1) passes.--all-targets --all-features, with Zig 0.15.2) passes.bun run check(with privacy:check) andbun run --cwd packages/cli typecheckpass.origin/next..HEADfinds no leaks.bun run test:local(cargo 1.97.1, no added load) at 8d591db: all 21 test binaries passed, including the lib (1,068 passed). Hosted CI at 8d591db is all green: test shards 1-4, Candidate core (ubuntu and macos), docker-e2e shards 1-4, Config compiler on 4 platforms, Runtime state models, linux-process-lifetime, runtime-images and secret-scan.Gaps
https_recovery::full_legacy_command_childfailed once in 30 loaded runs, withhttps_recovery_refusedafter 1.3 s. The fixture binds an ephemeral port, drops it, and later requiresport_absenton that port, so another socket on the host can take the port in between. This is a port-reservation race, not a timing bound, so it is reported as a finding and not changed here. The refusal is closed (no stage is reported), so I have not confirmed which check refused.host_endpoint,relay_loopandhttps_recoverytests use 1-3 s waits). None of them failed in the 60 loaded runs, so they were left as they are.STOP_HANDLE_WAIT(5 s) is unchanged.admission_pool.rsdocumentslsoftaking 5.4-6.5 s at load 30-80, sodownandrecoveron a loaded host can still fail asprovider_timeout_uncertain. This is reported as a finding and not changed here, because it changes product behavior.network_update::stoppedstill has a fixed, unnamed 5 slsofbudget. No lib test reaches it after the scan, so it is not load-sensitive in the suite. It is left as is.🤖 Generated with Claude Code