Skip to content

fix(native): keep on-failure restart-backoff and recheck-placeholder members unready until the readiness deadline - #362

Draft
roodboi wants to merge 1 commit into
nextfrom
wf/g11-restart-races-b18
Draft

roodboi wants to merge 1 commit into
nextfrom
wf/g11-restart-races-b18

Conversation

@roodboi

@roodboi roodboi commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Root cause

native-config-process-policy still fails on PRs that already contain #350. In run
39436132066 (PR #342, job 118673527798), hack --profile exercise up returned exit 1.
The job's value-free replay capsule shows after-compose observation 3:

  • Startup and strict both refused with E_NATIVE_COMPOSE_OWNERSHIP / topology /
    live-endpoint-membership, after 6 consumed replies (the first scan).
  • The bridge member keys were {hex64: 3, epPrefixed: 0, other: 0}, and no service was
    inspected as restarting.
  • A few seconds later, the startup diagnostic shows all four members present, retry
    running with restartCount: 2, and drift none.

Only retry (on-failure, max 2) restarts in this fixture. Here is what Docker does
on an on-failure restart. When the container exits non-zero and the policy allows
another attempt, the daemon releases its network endpoints during exit cleanup. It then
waits a backoff (100 ms, doubling per quick failure) and starts the same container ID
again. During the backoff, docker network inspect lists no key for the container.
On restart, Docker recreates the endpoint before the sandbox joins, so for a moment the
key is ep-<endpoint>. After the join, the key is the container ID again.

queryNativeComposeOwned reads containers, then volumes, then networks. Docker does
not lock these reads together. That leaves two races:

  • Case 2 (captured): retry is inspected running, then exits before the bridge
    read. The bridge has no key for it, so requireLiveMember refused it.
  • Case 1: retry is an ID member in the first scan, then exits and restarts before
    the recheck. The recheck reads its ep- placeholder, and networkMembers refused
    every recheck placeholder.

Both are documented gaps in #350.

Fix

In src/lib/native-compose-ownership.ts:

  • Startup readiness has a new membership mode, transient, used by every
    readiness-loop observation.
    • In this mode, requireLiveMember keeps a container unready when it was inspected
      running and is absent from its bridge (case 2).
    • networkMembers admits an ep- placeholder in the recheck scan (case 1). The
      container must have been inspected running or restarting, and the first scan
      must have seen the same container ID, name and service live, as an ID member or as
      absent, never as a placeholder.
  • Every existing condition still applies: a selected on-failure service, not a oneoff,
    exactly one owned container with that name, no ID membership, one placeholder per
    container, an endpoint on that network, the single owned outbound default bridge, and
    a matching network ID. Admission only makes the observation unready, so it never
    grants effect authority.
  • Shapes that still refuse exactly as before:
    • a placeholder that persists from the first scan into the recheck;
    • created, exited, unselected, oneoff, foreign, duplicate and malformed shapes;
    • every strict and saved read.
  • The new waitNativeComposeStartupOwned bounds the window with the existing
    readiness deadline. waitReady in src/lib/native-compose-command.ts now calls it.
    • If the deadline passes while the last observation was unready only because of
      these running shapes, it runs one settled query.
    • The settled query refuses with the original predicate (live-endpoint-membership
      or network-member-id) when the member is still missing or placeholdered.
    • It never turns a timeout into readiness.
    • A container inspected restarting keeps its earlier behavior: unready until the
      deadline, then the operation ends with an uncertain outcome.
  • The Docker restart-backoff semantics are documented on networkMembers and
    requireLiveMember.

Tests

  • New tests/native-process-policy-initial-trace.test.ts, case 2 (HACK-1279): the
    capsule's fixed fields are embedded. The test replays the captured first scan (retry
    running, absent, three hex64 members) through the real replay driver, with each
    recheck Docker can produce: rejoined, still in backoff, placeholder while joining, and
    still absent while read running.
    • On current next, startup returns exactly the capsule refusal:
      live-endpoint-membership, consumed 6, {hex64: 3}, restartingServices: [].
    • With this change, startup is unready with all 12 replies consumed.
    • Strict still equals the capsule refusal.
    • A new replay mode, settled, refuses a persistent absence with the capsule's
      startup fields and owns a member that has rejoined.
  • New trace test, case 1: an ID member in the first scan and an ep- key in the recheck
    (read running or restarting), plus case 2 progressing to a placeholder.
    • On current next, startup refuses with network-member-id at consumed 9.
    • With this change, startup is unready.
    • Settled and strict refuse.
    • A placeholder that persists across both scans still refuses at 9.
  • New tests/native-compose-ownership.test.ts:
    • "a placeholder first seen in the recheck scan stays unready only for the exact
      restarted container": unready in startup; refused in strict, saved and settled.
      Controls in the recheck that still refuse with network-member-id: created,
      exited, unknown-name, missing-name, non-record, also-member, duplicate,
      other-prefix, no-endpoint, unselected, oneoff, first-scan exited, and a first-scan
      placeholder. The fake Docker gained a recheck row override for this.
    • "startup wait bounds a missing running member by the readiness deadline":
      • A permanently missing member is unready until the deadline, then refuses with
        live-endpoint-membership.
      • A member that rejoins is ready before the deadline.
      • A restarting container still times out with null, as before.
      • An already-expired deadline reads nothing.
  • Updated pinned expectations that this change intentionally reverses: a running
    selected container with no membership and no placeholder is now unready in startup.
    It still refuses in settled and strict reads. Applies to the fix(native): admit a just-restarted on-failure placeholder in the first startup scan #350 test and to the
    "running" row of "restart observation keeps foreign and incomplete endpoint shapes
    fenced", which now checks every row in both modes.
  • tests/e2e/README.md describes the updated guard and the deadline bound.

Mutations

Each mutation was run against tests/native-compose-ownership.test.ts and
tests/native-process-policy-initial-trace.test.ts:

Mutation Result
M1: revert running-absent admission (case 2) 5 fail, including both HACK-1279 replays and the deadline test
M2: revert recheck-placeholder admission (case 1) 3 fail
M3: settled behaves as transient 5 fail
M4: recheck placeholder ignores the first scan 5 fail, including the persistent-placeholder controls
M5: the wait never runs the settled check 1 fail (the deadline test)
M7: settled admits a first-scan running placeholder 1 fail
M8: placeholder admission accepts a oneoff owner 2 fail
M6: strict reaches the startup admission survives

M6 is an equivalent mutant. Strict and saved reads pass an empty on-failure selection,
so restartingServices.has(...) still refuses them.

Validation

  • Bun 1.4.2 via mise x bun@1.4.2, after bun install --frozen-lockfile. There are no
    lockfile changes.
  • bun test passed 120/120 across tests/native-compose-ownership.test.ts,
    tests/native-process-policy-initial-trace.test.ts,
    tests/native-process-policy-replay-reason.test.ts,
    tests/native-compose-effect-diagnostics.test.ts,
    tests/cli-spawn-env-isolation.test.ts, tests/native-compose-wait-ready.test.ts,
    tests/native-compose-completion.test.ts and
    tests/native-config-process-policy.test.ts.
  • bun run --cwd packages/cli typecheck is clean. bun run --cwd packages/cli check
    reports 0 errors and 34 warnings, the same as the base.
  • Full bun run test ran on a heavily loaded host (load average about 70): 7058 pass,
    149 skip, 253 fail across 40 files. I reran each failing file serially with
    --timeout 60000:
    • 36 of the 40 files passed.
    • Four files still fail. All four also fail with the same source files checked out
      from origin/next, and none of them import the changed modules:
      • tests/native-authored-project-start.test.ts (a startup fixture cascade at stage
        runtime; 42 fail on base, 40 here)
      • tests/native-authored-project-recovery.test.ts (the same 2 tests)
      • tests/test-config-compiler.test.ts ("timeout reaps the owned compiler")
      • tests/native-compose-vm-file-prepending.test.ts (45 s timeouts in a different
        case on each run)
  • gitleaks git --log-opts=origin/next..HEAD reports no leaks.
  • I did not run the docker-e2e scenario locally. The hosted docker-e2e on this PR is the
    runtime evidence.

Remaining gaps

  • The settled check runs after the readiness deadline, so a refusal can arrive up to one
    probe timeout after it.
  • With this change, a permanently disconnected running on-failure container is no
    longer refused in the first query. It now refuses when the readiness deadline expires.

Refs HACK-1279

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

…members unready until the readiness deadline

The startup Compose readiness check refused two Docker on-failure restart races
that #350 left open:

- Case 2 (captured in run 39436132066, after-compose observation 3): the retry
  container was inspected running, then exited, and Docker released its
  endpoint for the restart backoff before the same scan read the bridge. The
  member was absent with no placeholder, so startup refused with
  live-endpoint-membership.
- Case 1: the retry container was an ID member in the first scan, then
  restarted, and the recheck read its new ep- placeholder. Startup refused with
  network-member-id.

Startup now keeps both unready, but only for a selected, non-oneoff on-failure
container read running or restarting, on the single owned default bridge. A
recheck placeholder is admitted only for the same container that the first
scan saw by ID or as absent. A placeholder that persists across both scans,
and every foreign, malformed, strict or saved shape, still refuses as before.

waitNativeComposeStartupOwned bounds this with the existing readiness
deadline. If the deadline passes while the last observation was unready only
because of these running shapes, one settled check refuses a member that is
still missing or placeholdered.

Refs HACK-1279

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 20, 2026

Copy link
Copy Markdown

HACK-1279

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant