fix(native): re-observe a sandbox-less bridge endpoint instead of refusing adopted Compose (HACK-1421) by roodboi · Pull Request #477 · hack-dance/hack

roodboi · 2026-10-24T06:05:56Z

The docker e2e scenario native-compose-adoption-job-worktrees failed once on next (run 40055366376, job 120757063141, docker-e2e shard 4/4, commit d803ad6). At phase=alpha-start-7, hack up exited with code=E_CONFIG_INVALID ordered-stage=unavailable after about 2.8 s. Passing runs of the same up take 7 to 11 s.

Root cause

The task suggested that #452 had started mapping input access faults to E_CONFIG_INVALID. That is not the case. #452 only added a value-free diagnostic and changed one message. ProjectInputAccessError already returned E_CONFIG_INVALID before it. The scenario also has no concurrent writer, so I found no half-written .hack input to blame.

The cause I found is a Docker read race in the retained-resource binding proof (src/lib/native-compose-adoption-binding.ts). I found it by reading the source. The CI log does not show it directly (see Gaps).

While an adopted up runs its ordered job plan, it keeps calling assertFresh(). Each call runs readInputs() and then inspectResources(). That function reads each owned bridge's .Containers keys with docker network inspect.
Docker lists an endpoint that is not joined to a sandbox as ep-<endpoint id>. Moby's buildContainerAttachments (daemon/network.go) does this, and it happens for a moment on every container start (endpoint created, sandbox not yet joined) and exit (sandbox left, endpoint not yet deleted). A container inspect waits for the container lock, but a network inspect does not, so it can see this key. The native Compose owner already handles this key (fix(native): re-observe restarting placeholder network members #272, fix(native): keep on-failure restart-backoff and recheck-placeholder members unready until the readiness deadline #362). The adoption binding did not.
networkRow and pluralNetworkRows accepted only 64-hex container IDs, so the key failed at once with E_LEGACY_COMPOSE_BINDING_IDENTITY. The snapshot loop retries only when running state changes, so this refusal was never retried. Inside the job plan the error carries no ordered-refusal tag, so the store returns E_LEGACY_ADOPTION_STATE and the command reports E_CONFIG_INVALID with no legacy_adoption_refusal detail. That is exactly what the CI line shows.
At alpha-start-7 the DB is already healthy, so the seed job starts within the first seconds of the up, during the polling. This fits the early 2.8 s refusal. The window is only milliseconds long, so the failure is rare (once in the last 100 ci.yml runs).

Fix

A new helper, bridgeMembers(), splits a bridge's keys into container IDs and a transient flag. A key that exactly matches ep-<64 hex> is never attributed, admitted or saved. Any other non-ID key, or a duplicate, still refuses identity at once.
If the accepted loop read has a placeholder, the existing snapshot loop throws that observation away and reads again. It keeps the same aggregate probe deadline and output allowance. A placeholder that is still there at the deadline refuses with the probe deadline error. The first bridge read is only an identity read and is always followed by a loop read, so a placeholder there costs no extra scan.
docs/reference/native-compose-adoption.md now describes this rule next to the existing note that "delayed endpoint removal can still refuse".
Diagnostic: the scenario's cli-result line now ends with refusal=<token>, for example refusal=legacy-state. The token comes from a closed list. It is set only when the envelope message exactly matches one of the fixed product messages: the LegacyComposeAdoptedGenerationError codes, the adopted-command refusal and cancel messages, the adoption marker kinds, and the shared input-access guard. Any other message gives other. The message text is never printed. E_PERMISSION is now also a known code.

The task asked to surface #462's errno diagnostic in the scenario output. That is not possible: #462 is still open, and its diagnostic is internal and never reaches the CLI envelope. The new token does tell an input-access or marker fault apart from an adoption-store refusal.

Tests and mutation evidence

New tests in tests/native-compose-adoption-binding.test.ts. The fake docker adds an ep-<hex> key to a chosen network inspect read:

placeholder on the identity read (no extra scan) and on the accepted loop read (exactly one extra scan): the binding equals a clean acquisition.
placeholder on one of two owned bridges: both bridges are read again.
a placeholder on every read uses up the shared deadline and refuses with E_LEGACY_COMPOSE_BINDING_PROBE.
a malformed key that looks like a placeholder (ep-not-an-endpoint) still refuses identity after one read.

Before the fix, the three placeholder cases and the deadline case failed with E_LEGACY_COMPOSE_BINDING_IDENTITY. That is the CI symptom.

I applied each mutation, ran the tests, and restored the file:

Mutation	Tests that failed
Drop `transient
Accept only IDs in `bridgeMembers` (the old behavior)	4: all placeholder cases and the deadline case
Loosen the pattern to `/^ep-/`	1: the malformed-key test
Drop the plural `transient
Diagnostics: map `legacy-state` to the CHANGED message	1: the new token test
Diagnostics: print the raw message instead of a token	4: every test that checks for leaked text
Diagnostics: match by prefix instead of exact text	1: the new token test (suffix case)

Validation

tests/native-compose-adoption-binding.test.ts: 104 pass, 0 fail. I ran it with --timeout 60000, a private HOME, HACK_HOME and DOCKER_CONFIG, and HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK=1. The run excluded one existing test, "moving job snapshot counterexample". It times out at its own 30 s limit on this Mac (load average 50 to 90), and it fails the same way with the unmodified origin/next source.
tests/native-compose-adoption-job-diagnostics.test.ts: 8 pass. The adoption execution, jobs, probe and mount-format files also pass. tests/native-compose-adoption-job-worktrees.test.ts has 1 or 2 process-timing failures on this loaded Mac ("Synthetic owned child did not become ready" in 2 s). They fail the same way with the original diagnostics file.
bun run --cwd packages/cli typecheck: 0 errors.
Root bun run check: privacy-check: ok. @hack/cli:check shows the 34 existing warnings. In the binding module, the same 4 complexity warnings existed before; inspectResources goes from 30 to 31.
docs:cli-reference and generate:agent-plugins are not needed: the CLI surface and the consumer agent guidance did not change.
gitleaks 8.30.1 (gitleaks dir on the exported git log -p origin/next..HEAD): no leaks found.
I ran no Docker e2e on this Mac. Hosted CI at head 3d26b784 (run 40068851447): all four docker-e2e shards and the aggregate docker-e2e gate pass. native-compose-adoption-job-worktrees passes in shard 2 (98.3 s), and its cli-result lines now end with refusal=. The Config compiler, Candidate core (ubuntu), runtime-images, linux-process-lifetime, Runtime state models and secret-scan checks also pass.
Hosted test is red: shard 3/4 failed one test in tests/project-env-layer-lock.test.ts, "concurrent legacy dotenv sets and removals of different keys all apply". A concurrent upsertDotEnvValue read the env file during a replacement and refused it as "a symlink or not a regular file". This PR does not touch that code. HACK-1422 already tracks this as an intermittent failure. It is not on this workflow's list of flakes that may be rerun, so I did not rerun it. Shards 1, 2 and 4 pass.

Gaps

The CI log does not name the cause. The scenario printed only the code and ordered detail, no artifacts were kept, and the CLI gives no finer stage. The ep- race is the only path I found that matches all the evidence. But it is inferred from the source and Moby's code, not observed. If the failure comes back, the new refusal= token will show whether it is an adoption-store refusal (legacy-state) or something else.
legacy-state is still shared by many store sites. A recurrence with legacy-state after this fix would point to another transient read, such as ID-keyed delayed endpoint removal, which the doc already lists as able to refuse.
I did not reproduce the window against a real daemon, by design (no local Docker e2e).

Refs HACK-1421

Round 2 (current head `c9ce3f9`)

Merged origin/next 6e4bcf2 into the branch (no conflicts; the diff against next is unchanged).
Hosted CI run 40102249794 at c9ce3f9: every check passed, including all four test shards and the aggregate test, and all four docker-e2e shards. The earlier red test at 3d26b78 (run 40068851447) was HACK-1422's legacy dotenv concurrency test, which this PR does not touch; its fix is fix(env): retry a private env read that races an atomic replace instead of refusing it as a symlink #480.
Independent verification: next's binding module fails 4 of the 5 new placeholder tests (3 with E_LEGACY_COMPOSE_BINDING_IDENTITY, the CI symptom); the PR head passes them. Mutations of the binding and diagnostics code were killed.
Remaining follow-ups (hand-copied diagnostic strings, a load-sensitive two-bridge test, a pre-existing untested duplicate-key check) are tracked separately.

🤖 Generated with Claude Code

…using adopted Compose (HACK-1421) Docker lists an endpoint that is not joined to a sandbox as `ep-<endpoint>` in a bridge's `.Containers`, for a moment on every container start and exit. The bridge read takes no container lock, so the retained-resource binding proof could see that key while a job or service changed state. The member check accepted only 64-hex container IDs, so it refused the binding identity immediately. Inside an adopted start, that refusal surfaces as a closed E_CONFIG_INVALID with no ordered-refusal detail, which matches the flaky native-compose-adoption-job-worktrees failure at alpha-start-7. A placeholder member is now never attributed, admitted or saved. It marks that bridge read as transient, and the existing snapshot loop discards the observation and re-reads under the same aggregate deadline and output allowance. A placeholder that persists until the deadline still refuses, and any other non-ID key still refuses identity at once. The completed-job scenario's cli-result line also names a fixed refusal message by closed token (refusal=legacy-state, legacy-changed, input-access, ...), so a recurrence says which owner refused without any value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

linear-code · 2026-10-24T06:06:02Z

HACK-1421

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…udgets for load (HACK-1432) (#487) Follow-ups from #477 (HACK-1421). Test and diagnostic hardening only: the adopted command's behavior and messages do not change. ## Root cause 1. **Diagnostic drift.** `tests/e2e/scenarios/native-compose-adoption-job-diagnostics.ts` maps the adopted command's fixed messages to closed tokens by exact match, but the `adopted-refused` and `adopted-cancelled` messages were hand-copied literals. A wording change in `src/lib/native- compose-adoption-command.ts` would silently turn either token into `other`. No test covered `adopted-cancelled`. 2. **Load-sensitive two-bridge test.** "a placeholder on one of two owned bridges is re-observed with both bridges" does two extra bridge scans inside one probe, which has a default 15 s deadline (`createNativeComposeProbe`). Under heavy load the probe refused with `E_LEGACY_COMPOSE_BINDING_PROBE`. 3. **Untested duplicate-key check.** No test fed `bridgeMembers()` a duplicate `.Containers` key, so the `new Set(value).size === value.length` check could be removed without any test failing. 4. **5 s timeouts.** Most tests in `tests/native-compose-adoption-binding.test.ts` ran with Bun's 5 s default, and some had ad hoc 10-30 s overrides. Every fake Docker query is a cold Bun process, so under load those budgets ran out while the host was still starting processes, before the product logic finished. The 30 s snapshot case is worse: when it times out it leaves `snapshotCaseUnconfirmed` set, and every later test in the file then fails in `beforeEach`. ## Fix - `src/lib/native-compose-adoption-command.ts` exports `LEGACY_COMPOSE_ADOPTED_COMMAND_MESSAGES` (`refused`, `cancelled`). Both throw sites use it, and the scenario's token table imports it. - `tests/native-compose-adoption-job-diagnostics.test.ts`: one test per token (`adopted- refused`, `adopted-cancelled`) replaces the copied literal. - `tests/native-compose-adoption-binding.test.ts`: - `PROBE_DEADLINE_MS = 15_000` records the product's default probe deadline. - `FAKE_DOCKER_TEST_TIMEOUT_MS = 8 * PROBE_DEADLINE_MS` is set once with `setDefaultTimeout`, replacing the 5 s default and the 10/20/30 s overrides. On a slow host, a test should end with the product's own closed refusal, not a Bun timeout. - Both placeholder re-observation tests pass `timeoutMs: PLACEHOLDER_RESCAN_PROBE_TIMEOUT_MS` (3 x default) to the clean baseline acquisition and to the placeholder acquisition. Under load, the clean baseline also refused at the 15 s default. Their test budget, `PLACEHOLDER_RESCAN_TEST_TIMEOUT_MS`, is the sum of their four probe deadlines. - New `test.each`: a duplicate container ID and a duplicate endpoint placeholder each refuse with `E_LEGACY_COMPOSE_BINDING_IDENTITY` on the first bridge read, with no retry (exactly one network inspect). - Deadline assertions that already existed (for example, the 3 s and 6 s shared-deadline cases) are unchanged. ## Tests and mutation proof Mutation runs are scripted. Each one applies an exact-text change, runs the focused tests, and restores the file. | Mutation | Result | | --- | --- | | Remove the scenario's `adopted-cancelled` entry | cancelled-token test fails | | Swap the two message constants in the scenario table | both token tests fail | | Scenario `adopted-cancelled` reverts to a drifted literal (`canceled`) | cancelled-token test fails | | Scenario `adopted-refused` reverts to a drifted literal (`kept`) | refused-token test fails | | `bridgeMembers()` duplicate check replaced by `true` (M6) | both duplicate-key tests fail: the ID case does an extra network read, and the placeholder case re- observes until the probe deadline | Focused runs: - `tests/native-compose-adoption-job-diagnostics.test.ts`: 10 pass, 0 fail. - Duplicate-key tests on the unmutated tree: 2 pass. ### Load comparison (item 4) This host has 18 cores and is shared with other agents. I ran `tests/native-compose-adoption- binding.test.ts` from an extracted `origin/next` tree (1281184) and from this branch **at the same time**, so both saw the same load. Each run used a private HOME, HACK_HOME and DOCKER_CONFIG, with keychain fallback disabled. | Round | Load (1/5/15 min) at start | unmodified next | this PR | Bun timeouts (next / PR) | | --- | --- | --- | --- | --- | | 1 (separate runs, before the second commit) | about 30 (next); rose from about 30 to 90 during the PR run | 4 pass / 101 fail / 1 error | 102 pass / 5 fail / 1 error | 2 / 0 | | 2 (before the second commit) | 33 / 68 / 81 | 1 pass / 104 fail | 97 pass / 10 fail | 5 / 0 | | 3 (final head content) | 33 / 32 / 46 | 3 pass / 102 fail | 98 pass / 9 fail | 3 / 0 | On next, a few tests hit Bun's 5 s timeout, and the 30 s snapshot case times out. That leaves `snapshotCaseUnconfirmed` set, so every later test fails in `beforeEach`. On this branch there are no Bun timeouts and no cascade. The failures that remain under this load are the product's own closed refusals, or timing assertions that are already in the file: - 6 tests: `E_LEGACY_COMPOSE_BINDING_PROBE`, because one acquisition took longer than the product's 15 s default probe deadline. Those tests do not name a deadline. - The snapshot case ends in the 20 s operation deadline's ordered refusal (closed and value- free) instead of a Bun timeout. - Elapsed-time and child-reaping assertions: the 3 s shared-deadline case, "captured cancellation ...", and "cancelled real child ...". I have not changed these. They are listed as findings. ## Gates - gitleaks 8.30.1 (`gitleaks dir` over a `git log -p origin/next..HEAD` export): no leaks. - Root `bun run check` (includes `privacy:check`): passes. The 34 warnings are existing complexity warnings in untouched files. - `bun run --cwd packages/cli typecheck`: 0 errors. - Biome check on the four changed files: clean. - No CLI surface or consumer agent guidance changed, so no docs or plugin regeneration. - Docker e2e was not run locally (`HACK_E2E_DOCKER` unset). The scenario module change is covered by the unit test that imports it, and hosted docker-e2e shards run it. ## Gaps - Hosted CI run 40112117595 at exact head a419882 passed: test shards 1-4, docker-e2e shards 1-4, candidate core, config compiler, runtime state models, linux-process-lifetime, runtime- images and secret-scan. No reruns. - Item 4: under the load on this host (load 30 to 90, 18 cores), the file has no Bun timeouts on this branch, but it does not fully pass. About 6 tests still exceed the product's 15 s **probe** deadline, and a few timing assertions fail. Widening every acquisition's probe deadline would touch about 60 call sites and stop exercising the default deadline, so it is left out of this PR and reported as a follow-up. - I did not re-measure at the issue's lower load band (18 to 50) separately from the rounds above. Round 3 started at load 33. - Docker e2e was not run locally (by design). The completed-job scenario that imports the new constant runs in the hosted docker-e2e shards. ## Findings (not fixed here) - **Default probe deadline under load:** at host load 30 or more, about 6 binding tests that use the product's default 15 s probe deadline refuse with `E_LEGACY_COMPOSE_BINDING_PROBE`. Possible follow-up: give those tests a named, wider probe deadline across the file. - **Fixed start-up wait:** "cancelled real child is reaped and abort reason stays private" waits only 100 x 5 ms for the fake Docker child to start. Under load the test fails, and its abandoned acquisition then rejects later as an "Unhandled error between tests". - **Child still alive after cancellation:** "captured cancellation cannot be replaced by a later fresh signal" saw the cancelled probe child's PID still alive (`process.kill(pid, 0)` did not throw) right after the `E_LEGACY_COMPOSE_BINDING_CANCELLED` refusal under load. Either cancellation can return before the child is reaped, or the assertion is racy. This needs investigation. Refs HACK-1432 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Merge branch 'next' into wf/g86-adoption-job-worktrees-config-invalid

c9ce3f9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

roodboi mentioned this pull request Oct 24, 2026

train b54 integration gate (do not merge) #483

Closed

roodboi marked this pull request as ready for review October 24, 2026 15:46

roodboi merged commit 1281184 into next Oct 24, 2026
21 checks passed

roodboi deleted the wf/g86-adoption-job-worktrees-config-invalid branch October 24, 2026 15:47

roodboi mentioned this pull request Oct 24, 2026

test(native): import adopted refusal messages and size binding test budgets for load (HACK-1432) #487

Merged

Navigation Menu

Repository navigation

fix(native): re-observe a sandbox-less bridge endpoint instead of refusing adopted Compose (HACK-1421) - #477

fix(native): re-observe a sandbox-less bridge endpoint instead of refusing adopted Compose (HACK-1421)#477
roodboi merged 2 commits into
nextfrom
wf/g86-adoption-job-worktrees-config-invalid

roodboi commented Oct 24, 2026 •

edited

Loading

Uh oh!

linear-code Bot commented Oct 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

roodboi commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root cause

Fix

Tests and mutation evidence

Validation

Gaps

Round 2 (current head c9ce3f9)

Uh oh!

linear-code Bot commented Oct 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

roodboi commented Oct 24, 2026 •

edited

Loading

Round 2 (current head `c9ce3f9`)