Repository navigation
Conversation
…members unready until the readiness deadline The startup Compose readiness check refused two Docker on-failure restart races that #350 left open: - Case 2 (captured in run 39436132066, after-compose observation 3): the retry container was inspected running, then exited, and Docker released its endpoint for the restart backoff before the same scan read the bridge. The member was absent with no placeholder, so startup refused with live-endpoint-membership. - Case 1: the retry container was an ID member in the first scan, then restarted, and the recheck read its new ep- placeholder. Startup refused with network-member-id. Startup now keeps both unready, but only for a selected, non-oneoff on-failure container read running or restarting, on the single owned default bridge. A recheck placeholder is admitted only for the same container that the first scan saw by ID or as absent. A placeholder that persists across both scans, and every foreign, malformed, strict or saved shape, still refuses as before. waitNativeComposeStartupOwned bounds this with the existing readiness deadline. If the deadline passes while the last observation was unready only because of these running shapes, one settled check refuses a member that is still missing or placeholdered. Refs HACK-1279 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
native-config-process-policystill fails on PRs that already contain #350. In run39436132066 (PR #342, job 118673527798),
hack --profile exercise upreturned exit 1.The job's value-free replay capsule shows after-compose observation 3:
E_NATIVE_COMPOSE_OWNERSHIP/topology/live-endpoint-membership, after 6 consumed replies (the first scan).{hex64: 3, epPrefixed: 0, other: 0}, and no service wasinspected as
restarting.retryrunning with
restartCount: 2, and driftnone.Only
retry(on-failure, max 2) restarts in this fixture. Here is what Docker doeson an on-failure restart. When the container exits non-zero and the policy allows
another attempt, the daemon releases its network endpoints during exit cleanup. It then
waits a backoff (100 ms, doubling per quick failure) and starts the same container ID
again. During the backoff,
docker network inspectlists no key for the container.On restart, Docker recreates the endpoint before the sandbox joins, so for a moment the
key is
ep-<endpoint>. After the join, the key is the container ID again.queryNativeComposeOwnedreads containers, then volumes, then networks. Docker doesnot lock these reads together. That leaves two races:
retryis inspectedrunning, then exits before the bridgeread. The bridge has no key for it, so
requireLiveMemberrefused it.retryis an ID member in the first scan, then exits and restarts beforethe recheck. The recheck reads its
ep-placeholder, andnetworkMembersrefusedevery recheck placeholder.
Both are documented gaps in #350.
Fix
In
src/lib/native-compose-ownership.ts:transient, used by everyreadiness-loop observation.
requireLiveMemberkeeps a container unready when it was inspectedrunningand is absent from its bridge (case 2).networkMembersadmits anep-placeholder in the recheck scan (case 1). Thecontainer must have been inspected
runningorrestarting, and the first scanmust have seen the same container ID, name and service live, as an ID member or as
absent, never as a placeholder.
exactly one owned container with that name, no ID membership, one placeholder per
container, an endpoint on that network, the single owned outbound default bridge, and
a matching network ID. Admission only makes the observation unready, so it never
grants effect authority.
waitNativeComposeStartupOwnedbounds the window with the existingreadiness deadline.
waitReadyinsrc/lib/native-compose-command.tsnow calls it.these
runningshapes, it runs onesettledquery.live-endpoint-membershipor
network-member-id) when the member is still missing or placeholdered.restartingkeeps its earlier behavior: unready until thedeadline, then the operation ends with an uncertain outcome.
networkMembersandrequireLiveMember.Tests
tests/native-process-policy-initial-trace.test.ts, case 2 (HACK-1279): thecapsule's fixed fields are embedded. The test replays the captured first scan (
retryrunning, absent, three hex64 members) through the real replay driver, with each
recheck Docker can produce: rejoined, still in backoff, placeholder while joining, and
still absent while read
running.next, startup returns exactly the capsule refusal:live-endpoint-membership, consumed 6,{hex64: 3},restartingServices: [].unreadywith all 12 replies consumed.settled, refuses a persistent absence with the capsule'sstartup fields and owns a member that has rejoined.
ep-key in the recheck(read
runningorrestarting), plus case 2 progressing to a placeholder.next, startup refuses withnetwork-member-idat consumed 9.unready.tests/native-compose-ownership.test.ts:restarted container": unready in startup; refused in strict, saved and settled.
Controls in the recheck that still refuse with
network-member-id: created,exited, unknown-name, missing-name, non-record, also-member, duplicate,
other-prefix, no-endpoint, unselected, oneoff, first-scan exited, and a first-scan
placeholder. The fake Docker gained a
recheckrow override for this.live-endpoint-membership.restartingcontainer still times out withnull, as before.runningselected container with no membership and no placeholder is now unready in startup.
It still refuses in settled and strict reads. Applies to the fix(native): admit a just-restarted on-failure placeholder in the first startup scan #350 test and to the
"running" row of "restart observation keeps foreign and incomplete endpoint shapes
fenced", which now checks every row in both modes.
tests/e2e/README.mddescribes the updated guard and the deadline bound.Mutations
Each mutation was run against
tests/native-compose-ownership.test.tsandtests/native-process-policy-initial-trace.test.ts:settledbehaves astransientrunningplaceholderM6 is an equivalent mutant. Strict and saved reads pass an empty on-failure selection,
so
restartingServices.has(...)still refuses them.Validation
mise x bun@1.4.2, afterbun install --frozen-lockfile. There are nolockfile changes.
bun testpassed 120/120 acrosstests/native-compose-ownership.test.ts,tests/native-process-policy-initial-trace.test.ts,tests/native-process-policy-replay-reason.test.ts,tests/native-compose-effect-diagnostics.test.ts,tests/cli-spawn-env-isolation.test.ts,tests/native-compose-wait-ready.test.ts,tests/native-compose-completion.test.tsandtests/native-config-process-policy.test.ts.bun run --cwd packages/cli typecheckis clean.bun run --cwd packages/cli checkreports 0 errors and 34 warnings, the same as the base.
bun run testran on a heavily loaded host (load average about 70): 7058 pass,149 skip, 253 fail across 40 files. I reran each failing file serially with
--timeout 60000:from
origin/next, and none of them import the changed modules:tests/native-authored-project-start.test.ts(a startup fixture cascade at stageruntime; 42 fail on base, 40 here)tests/native-authored-project-recovery.test.ts(the same 2 tests)tests/test-config-compiler.test.ts("timeout reaps the owned compiler")tests/native-compose-vm-file-prepending.test.ts(45 s timeouts in a differentcase on each run)
gitleaks git --log-opts=origin/next..HEADreports no leaks.runtime evidence.
Remaining gaps
probe timeout after it.
runningon-failure container is nolonger refused in the first query. It now refuses when the readiness deadline expires.
Refs HACK-1279
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.