Skip to content

test(runtime-core): keep dead relay owner fixtures uncleaned until SIGKILL reaps them - #473

Merged
roodboi merged 1 commit into
nextfrom
wf/g85-relay-owner-dead-recovery-flake
Oct 24, 2026
Merged

roodboi merged 1 commit into
nextfrom
wf/g85-relay-owner-dead-recovery-flake

Conversation

@roodboi

@roodboi roodboi commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Refs HACK-1420

Root cause

The dead-owner publication tests killed the fixture child and then called Child::wait. Child::wait closes the child's stdin before it reaps. In the fixture, stdin EOF is the signal to exit cleanly: publication_child returns, and ControlListener::drop unlinks the socket and then the receipt.

Under load, the child can wake on that EOF and run part or all of that drop before SIGKILL takes effect. The test then sees a publication that has changed. Recovery correctly refuses it, and the failure shows up as:

  • Os NotFound at tests.rs:299 (fs::remove_file(pin.paths.socket)), or Dead relay publication refused: receipt_unchanged=true socket_unchanged=false process_absent=true child_signal=Some(9) (the child was SIGKILLed, but only after it had removed its socket).
  • relay_owner_refused from dead::Witness::acquire at :384 and :432.

The product code was correct. Failing closed on a changed publication is the intended behavior. The race was only in the test fixture.

Fix (test-only)

A new kill_and_reap(child, pin) helper replaces kill() + wait() in all three tests:

  1. It takes the stdin write end and holds it, so SIGKILL is the child's only way out.
  2. It reaps with try_wait under the named REAP_BUDGET (10 s). Running past the budget fails with E_FIXTURE_REAP_TIMEOUT and does not continue.
  3. It asserts that the exit signal was SIGKILL (E_FIXTURE_OWNER_EXIT).
  4. It asserts that the receipt and the socket are still the exact pinned pair (E_FIXTURE_OWNER_CLEANED) before any dead-owner assertion runs.

None of the dead-owner assertions were removed or relaxed.

Evidence (macOS, cargo +1.97.1, private HOME/HACK_HOME/DOCKER_CONFIG, HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK=1)

The load was 8 concurrent loops of the filtered lib test binary plus 18 busy-loop CPU workers on an 18-core host. A trap stopped every load process.

Binary Filter Runs Failures
base (next before fix) relay_owner::publication 200 42 (all three tests; NotFound :299, refused :321, relay_owner_refused :384/:432)
base the three tests 400 39
fix relay_owner::publication 200 0
fix, rebased on 0ca0e45 the three tests 8 x 50 0 (each loop 50 consecutive passes)
fix, rebased on 0ca0e45 relay_owner::publication 200 0

A serial loop with load (60 runs of the three tests) passed on both base and fix, so it does not reproduce the race. The parallel loops above are the discriminating check.

Mutation proofs

  • M1: release stdin right after kill(), which is what Child::wait does. Result: 23/200 failures under load, all E_FIXTURE_OWNER_CLEANED. This shows that holding stdin is the fix, and that the post-reap publication check catches a cleanup race.
  • M2: set REAP_BUDGET = Duration::ZERO. Result: 3/3 runs failed in every test with E_FIXTURE_REAP_TIMEOUT. This shows the budget check is enforced.
  • M3: expect SIGTERM instead of SIGKILL. Result: every test failed with E_FIXTURE_OWNER_EXIT. This shows the exit-signal check is enforced.

Gates

  • bun run check:local (fmt + clippy, all targets, -D warnings): pass at the rebased head.
  • bun run test:local: pass on the pre-rebase head (lib 1047 passed, 60 ignored; all other targets ok). The rebase onto test(runtime-core): give test fixtures fresh, uniquely named temp roots #461 touches no relay_owner files. Hosted CI is the full-suite evidence at the exact head.
  • Root bun run check (includes privacy:check): pass.
  • bun run --cwd packages/cli typecheck: 0 errors.
  • gitleaks 8.30.1 over origin/next..HEAD: no leaks.
  • The relay_owner module is compiled only on macOS and has no Cargo feature gate, so an all-features clippy run adds nothing here.

Gaps

  • Hosted CI is green at the exact head 3fc82f0: Candidate core (macos-latest and ubuntu-latest), test, all docker-e2e shards, and every other check. Ubuntu does not compile this module, so the macOS job is the one that runs these tests.
  • The hosted macOS run is one pass, not a run under load. The repeated-run proof under load is local only.
  • Not run on the M3; this unit uses hosted CI for full-suite evidence.

🤖 Generated with Claude Code

…GKILL reaps them

The dead-owner publication tests killed the fixture child and then called
Child::wait, which closes the child's stdin before it reaps. Stdin EOF is the
fixture's clean-exit trigger, so a loaded child could wake on that EOF before
SIGKILL landed and run its ControlListener drop, unlinking the socket (and
sometimes the receipt). Recovery then correctly refused the changed
publication, which surfaced as NotFound at remove_file or relay_owner_refused
from verify_dead and dead::Witness::acquire.

kill_and_reap now holds the stdin write end until the child is reaped with
try_wait under a named REAP_BUDGET, asserts the exit was SIGKILL, and asserts
the receipt and socket are still the exact pinned pair before any dead-owner
assertion runs. Product recovery is unchanged.

Refs HACK-1420

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 24, 2026

Copy link
Copy Markdown

HACK-1420

@roodboi
roodboi marked this pull request as ready for review October 24, 2026 14:38
@roodboi
roodboi merged commit 5afbf3a into next Oct 24, 2026
17 checks passed
@roodboi
roodboi deleted the wf/g85-relay-owner-dead-recovery-flake branch October 24, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant