diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d80226ee5..1c5869a77 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -503,4 +503,4 @@ jobs: - name: Install dependencies run: bun install --frozen-lockfile - name: Run process and terminal regressions - run: bun test tests/daemon-command.test.ts tests/daemon-orphan.test.ts tests/host-exec-lifetime.test.ts tests/host-exec-tty.test.ts tests/shell-observation.test.ts tests/shared-host-lifecycle.test.ts + run: bun test tests/daemon-command.test.ts tests/daemon-orphan.test.ts tests/host-exec-lifetime.test.ts tests/host-exec-tty.test.ts tests/shell-observation.test.ts tests/shared-host-lifecycle.test.ts tests/project-shared-helpers.test.ts diff --git a/docs/lifecycle.md b/docs/lifecycle.md index e9761669a..1a4d90b2d 100644 --- a/docs/lifecycle.md +++ b/docs/lifecycle.md @@ -49,7 +49,7 @@ endpoint) and fails when it does not match. The [shared host helper lifecycle contract](./reference/shared-host-lifecycle.md) defines the identity-verified alternative: adoption only after the listener names the expected helper and upstream, exact PID-plus-start-time claims, serialized starters, and release that never signals a foreign -process. It is a library contract today; `singleton` does not use it yet. +process. Projects use it through [`lifecycle.sharedHelpers`](#shared-helpers); `singleton` is unchanged. ## Config @@ -182,6 +182,150 @@ Long-running processes live under `lifecycle.processes` and are objects with: Processes receive the resolved env contract (see `env.md`) as their environment. +### Shared helpers + +`lifecycle.sharedHelpers` declares host helpers that several branches (or projects) share +through one fixed loopback port, under the +[shared host helper contract](./reference/shared-host-lifecycle.md). Unlike `singleton`, a +listener is adopted only after it proves its identity, and Hack stops a helper only when it +started it and the last branch using it is released. + +```json +{ + "lifecycle": { + "sharedHelpers": [ + { + "name": "issuer-bridge", + "command": ["bun", "run", "bridge"], + "cwd": "tools/issuer-bridge", + "port": 4100, + "upstreamEnv": "ISSUER_UPSTREAM", + "env": ["ISSUER_TOKEN"] + } + ] + } +} +``` + +- `name` (required): helper kind. The helper must answer `GET http://127.0.0.1:` + with JSON `{"helper": "", "upstream": ""}` (an optional `pid` must be its own). +- `command` (required): argv array, executed directly, never through a shell string. +- `port` (required): the loopback port. `name` and `port` must be unique in the list. +- `upstream` or `upstreamEnv` (exactly one): the upstream this instance needs, literally or + from a variable in the resolved lifecycle env (then the CLI environment). Use + `upstreamEnv` with an env overlay when branches need different upstreams. +- `identityPath` (default `/__hack/identity`), `cwd` (repo-root relative), + `enabled` (default `true`), `readyTimeoutMs` (default 30000, at most 600000). +- `env`: variable names passed to the helper. It also gets `PATH`; nothing else from the + CLI environment is inherited. A listed name that is not set fails `hack up`. + +An invalid `sharedHelpers` entry is refused, not skipped: `hack up`, `hack down` and +`hack restart` fail with `E_LIFECYCLE_FAILED` before any hook, helper, lifecycle process or +Compose command runs. This is stricter than other config parse errors, which these commands +only warn about. Fix the entry and run the command again. + +Behavior: +- `hack up` and `hack restart` ensure each helper after `lifecycle.up.before` hooks and before + `lifecycle.processes`. Each instance (`` or `--`) is one holder. + A healthy helper is reused; an identity-matching helper someone else started is adopted; + otherwise Hack starts it and waits until its identity endpoint answers. +- A port held by anything else (another helper, the wrong upstream, a silent listener) fails + `hack up` with `E_LIFECYCLE_FAILED` and a closed reason; that process is left running. + Messages never include upstreams, argv or environment values. +- If `hack up` fails later, or receives SIGINT/SIGTERM at any point while helpers or + lifecycle processes start, Hack releases only the holders that run added and stops a + helper it had just started. Overlapping `hack up` runs of the same instance are ordered by + an instance lock; an earlier run's rollback never releases a holder after a later run of + that instance has started (the holder stays recorded until `hack down`). Holders of + containers that may still be running are kept instead (see below). +- A disabled helper (`enabled: false`) needs none of its `env` or `upstreamEnv` values. +- A project with `sharedHelpers` and a lifecycle session (lifecycle processes or a + persistent `up.before` hook) needs tmux for that session. With zellij selected, + `hack up` and `hack restart` fail with `E_LIFECYCLE_FAILED` before any hook, + helper or process runs: zellij records the session's ownership after creating + it, so an interrupted start could leave a session `hack down` cannot reclaim. + Install tmux or set `sessions.mux` to `tmux`. If the selection changes to zellij + while hooks run, the start fails when it chooses the session's backend, before + creating the session, and rolls back the helper holders it added. zellij + support is tracked in HACK-1849. +- `hack down` releases the instance's holder. A helper still used by another branch keeps + running; the last release stops a helper Hack started and never signals an adopted one. + Down releases every helper the instance acquired, recorded at start, so removing or + renaming an entry (or switching to a revision without it) does not strand its holder. +- A release that fails or cannot confirm the stop keeps the claim and fails `hack down` and + `hack restart` with `E_LIFECYCLE_FAILED` (non-zero exit, the helper names in `detail`). + Fix the claim and run `hack down` again. +- Holders are released only after the instance's containers and lifecycle session + stopped. `hack restart` keeps the containers running, so it keeps their holders and + reuses the helpers. Once a detached `hack up` or `hack restart` started the + containers, a later failure (such as a failing `up.after` hook) stops the lifecycle + processes but keeps the holders for `hack down`. If Compose fails or times out while + starting containers (including a failed restart repair), the holders are kept unless + `docker compose ps` confirms that no container of the instance may still run. + SIGINT/SIGTERM once a container start has begun keeps the holders too (the start may + still be starting services). If `docker compose down` fails, times out or is interrupted, the holders are + kept (with a warning) and the command fails with Compose's error; run `hack down` + again. If the session stop fails or cannot be confirmed (including when tmux cannot + say whether the session exists), the session's processes may still use the helpers: `hack + down`, `hack restart` and native retirement keep every holder, warn, and fail the same + way. Run `hack down` again once the session can be stopped. Likewise, an up whose failed + start kept its lifecycle session (its stop could not be confirmed) keeps the holders it + added. +- A `hack down` that overlaps a `hack up` of the same instance (same branch): + - `hack down` marks the instance as stopping once the down is committed: after its + `lifecycle.down.before` hooks succeed (and, on the native path, after the ownership and + finalization rechecks that follow them), right before Compose or graph shutdown. A down + refused before that point supersedes nothing. Its release + removes the instance's operation record. Both advance the instance's shutdown + generation. + - A down supersedes every `hack up` of the instance whose lifecycle preparation began + before it, even after a later `hack up` succeeds. Preparation begins just before the + `lifecycle.up.before` hooks run, so a down that completes while those hooks run also + supersedes that up: it fails with `E_LIFECYCLE_FAILED` before starting any helper. The + later up's helpers need not be the earlier up's (an entry may have been disabled or + changed), so the earlier up never counts as current again. + - **Prevented during preparation.** A `hack up` checks the instance right before its + containers start (after detached image preparation, before an attached `up`, and at the + end of native preparation). If a down marked or released it by then, that up fails with + `E_LIFECYCLE_FAILED` and starts no container. + - **Detected after launch, not prevented.** The check takes the instance lock only + briefly, so a down can still complete between the check and container start, or while + containers start. The up checks again after Compose returns: a detached up then fails + with `E_LIFECYCLE_FAILED` (its containers may already run); an attached up only checks + after Compose exits. In that case run `hack down` for the branch (`--branch `) to + stop the containers, then `hack up` again. +- Two overlapping `hack up` runs of the same instance, with no down in between: + - The newer up does not supersede the earlier one: an earlier, slower up still starts, + because the newer up holds the same helpers. + - The up that created the lifecycle session stops it on failure (or SIGINT/SIGTERM) only + if no other up adopted it since. Hack records each session adoption and creation under + the shared helper instance lock, so this holds whichever up ensured its helpers first. + An up that only adopted the session never stops it. + - An up whose lifecycle session another up replaced while it was still starting (its + entry was still pending) fails with an error saying so. It does not publish over the + replacement, start processes or set environment in it, or stop it; the replacement + stays owned and `hack down` can stop it. + - These guarantees hold for tmux commands that finish within their time limit + (15 s). Hack stops waiting for a command that does not, and treats its outcome + as unknown, but a stalled tmux server can still apply it later. A late kill + or window command from an earlier up can then stop or change a newer up's + session of the same branch; `hack down` and `hack up` recover it. A late + session creation leaves an empty session that the next `hack up` refuses; + stop it with tmux. Closing this gap is tracked in HACK-1849. + - A failed up releases only the helper holders it added, and only while it is the latest + up of the instance to have its helpers ready and no `hack down` has started since + (including one that was interrupted); otherwise the next `hack down` releases them. + - The failed up's other failure handling is unchanged (for example, it still clears the + instance's runtime-state entry), so check `hack ps --branch ` after such an + overlap. + - This applies to projects with `sharedHelpers`. If the shared helper instance lock stays + busy for 10 s (for example, a third up is still waiting for a slow helper), ownership + cannot be decided: the failed up keeps the lifecycle session and its helper holders, + warns, and leaves them for `hack down` to stop. +- Claims live under `~/.hack/shared-helpers/-/` (or `$HACK_HOME`), shared by + every project and branch that uses that helper name and port. +- Helpers are not part of the lifecycle mux session and do not appear in `hack logs`. + ## Visibility Lifecycle output is now surfaced across CLI/runtime views: diff --git a/docs/reference/shared-host-lifecycle.md b/docs/reference/shared-host-lifecycle.md index 587a8a172..5c0c9ae65 100644 --- a/docs/reference/shared-host-lifecycle.md +++ b/docs/reference/shared-host-lifecycle.md @@ -8,12 +8,14 @@ reuses and releases such a helper without killing a process it does not own. Source: [`src/lib/shared-host-lifecycle.ts`](../../src/lib/shared-host-lifecycle.ts). Fixtures: [`tests/shared-host-lifecycle.test.ts`](../../tests/shared-host-lifecycle.test.ts). -Status: this is a library contract with portable process fixtures. No configuration -key or CLI command uses it yet. Consumers (WU18 shared helpers, bounded external -claims and opt-in tunnels) adopt it in their own changes. The existing -`lifecycle.singleton` port check in [Lifecycle & startup](../lifecycle.md) is -unchanged. It adopts any listener on the configured ports; this contract adopts -only a listener that proves its identity. +Status: projects use this contract through `lifecycle.sharedHelpers` in +`hack.config.json` (see [Lifecycle & startup](../lifecycle.md#shared-helpers)): +`hack up` and `hack restart` ensure each helper, and `hack down` releases it. +Project wiring: [`src/lib/project-shared-helpers.ts`](../../src/lib/project-shared-helpers.ts). +Other consumers (bounded external claims and opt-in tunnels) adopt it in their own +changes. The existing `lifecycle.singleton` port check is unchanged. It adopts any +listener on the configured ports; this contract adopts only a listener that proves +its identity. ## States @@ -132,7 +134,10 @@ checkout. Every attempt first confirms that the claimed process is still live (`child-exited` otherwise). Cancellation (an `AbortSignal`) interrupts the probe and the backoff sleep. -7. On readiness, mark the claim `ready` and return `ready/owned`. On any failure of +7. On readiness, mark the claim `ready` and return `ready/owned`. Every `ready` + result carries `holderAdded`: whether this call recorded the holder. It is + decided under the lock, so a caller that rolls back a failed operation releases + only the holder that operation added. On any failure of a freshly spawned child (`deadline`, `cancelled`, `child-exited`, foreign answer), stop its birth-verified process tree and remove its claim. If the stop cannot be confirmed, keep the claim so the process stays recorded. @@ -267,6 +272,35 @@ pass is a receipt only once that hosted job has run it. | invalid claim / no census | Malformed, other-port, group-readable and oversized claims, and an unavailable census, fail closed; nothing is spawned or signalled; the claim is kept. | | release lock | A release while the lock is held times out with `lock-unavailable`; the claim and helper are unchanged. | | observer error | The observer's error is rethrown; the waiting gate exits and no claim remains. | +| holder added | A repeat ensure by the same holder reports `holderAdded: false`; a new holder reports `true`. | + +### Project and CLI receipts + +`bun test tests/project-shared-helpers.test.ts` covers config parsing, the claim +directory and holder, the helper environment and the start/release wiring with real +fixture processes; it is registered in the same hosted Linux job. +`bun tests/e2e/run.ts --only=shared-host-helpers` drives the +same helper through the CLI on two branches of one project (`hack up`/`down +--branch`, fake Docker, isolated `HACK_HOME`): + +| Phase | Asserted behavior | +| --- | --- | +| concurrent startup | Two `hack up --branch` runs in parallel: one owned helper, both holders recorded, the endpoint serves the upstream. | +| sibling with another upstream | `E_LIFECYCLE_FAILED` with `wrong-upstream` and no upstream value; the shared helper and its holders are unchanged. | +| first and last down | The first down retains the helper (still served); the last down stops it and removes the claim. | +| failed up | A Compose failure on branch A rolls back A's new holder only; branch B's holder and the helper stay. | +| child restart | A crashed owned helper is replaced under a fresh claim on the next up. | +| stale PID | A claim naming a live bystander PID with another birth is discarded; the bystander survives. | +| healthy shared helper | An identity-matching helper started elsewhere is adopted; down releases it and it keeps serving. | +| foreign port | A silent TCP owner fails `foreign-listener`, survives and leaves no claim. | +| shutdown during retry | SIGTERM to `hack up` while a never-ready helper is retried: the CLI exits 143 within 5 s, the owned child is stopped and the claim removed. | +| lost parent | SIGKILL to `hack up` leaves the helper recorded; the next up stops it at its deadline and a later up starts fresh. | +| down during a paused up | `hack up` is paused at Compose image preparation after its helpers are ready; `hack down` completes and stops the helper; the resumed up fails `E_LIFECYCLE_FAILED` and never runs `compose up -d`. | +| entry removed after start | After `hack up`, the `sharedHelpers` entry is removed from config; `hack down` still releases the recorded helper and stops it. | +| overlapping ups | A newer up adopts the session and helpers of an earlier up paused at image preparation. The earlier up then succeeds, fails in Compose, receives SIGTERM, or receives SIGTERM while the instance lock is held past its cleanup timeout; in every case the session, its ownership token and the helper survive. | +| down during up.before | An up waits in an `up.before` hook; a down completes; the up then fails `E_LIFECYCLE_FAILED`, starts no helper and runs no `compose up`. | +| disabled helper after down | An up is paused, a down completes, the helper is disabled, and a newer up succeeds; the earlier up still fails `E_LIFECYCLE_FAILED` and never runs `up -d`. | +| unreleased helper | With an unreadable claim, `hack down` exits non-zero with `E_LIFECYCLE_FAILED` naming the helper; the claim and helper are kept; after the claim is restored, down stops it. | The pull request records the source mutations used as negative controls and which test each one fails. Each test above that guards a single check was confirmed to @@ -287,7 +321,22 @@ fail when that check is removed. release would read the claim as stale and clear it without stopping the helper. A drift during a stop is handled: it gives `stop-unconfirmed`, never `stopped`. macOS records the start time directly and is not affected. -- Wiring into `hack.config.json`, `hack up`/`down` or the daemon, and migrating - existing `lifecycle.singleton` configurations. +- Model coverage. `tests/models/tla/shared-helper-instance` checks lifecycle-session + ownership across overlapping ups and a down (see its README section); the claim + protocol itself is covered by the fixtures above, not by a model. That model + assumes a timed-out tmux command, whose client Hack killed, has no later effect. + The code does not guarantee that: a stalled tmux server can still apply it, so a + late kill or window command can reach a newer session of the same instance + (recovered by `hack down` and `hack up`, not prevented). The README lists each + case. A session-ID mutation fence for tmux would close it; it is tracked in + HACK-1849 and is not in this slice. +- zellij lifecycle sessions for projects with shared helpers. They are refused + before anything starts: zellij writes the ownership token after creating the + session, so an interrupted start could leave a session without a token. + Support is tracked in HACK-1849. +- The daemon. It has no lifecycle surface today; shared helpers are started and + released only by the CLI's `up`, `restart` and `down`. +- Migrating existing `lifecycle.singleton` configurations. `singleton` keeps its + port-only adoption; moving a helper to `sharedHelpers` is an opt-in config change. - Native (macOS VM) and container-runtime effects. These fixtures qualify the process and claim protocol only. diff --git a/docs/reference/v5-model-coverage.md b/docs/reference/v5-model-coverage.md index 51241dd9d..744099816 100644 --- a/docs/reference/v5-model-coverage.md +++ b/docs/reference/v5-model-coverage.md @@ -42,6 +42,7 @@ must be attached to the eventual exact integration revision. | Data lifetime independent of compute; replaced/aliased metadata | native-persistent-enrollment, native-storage-witness, native-file-material | Existing-only enrollment, exact birth/witness and reference fencing; copied-label/replacement tests | Real SQL/content retention, external writer assumptions, allocated-byte evidence. | | Settled retained Compose route-attempt retirement; pinned references; interrupted deletion | **Coverage gap:** filesystem journal compaction is not modeled; native-route-stop covers a different stop protocol | Instance mutation lock and exact surviving claim witnesses in [journal compaction](../../src/lib/native-compose-route-claims.ts); [claim regressions](../../tests/native-compose-route-claims.test.ts) and [retained lifecycle integration](../../tests/native-compose-adoption-routing-generation.test.ts) | 24 successful operations keep two attempt directories and constant Linux descriptors; four integrated lifecycle operations check directory counts. Injected unlink failures cover same-store and fresh-store cleanup retries. Assumes cooperative same-user writers, the caller's mutation lock and working filesystem synchronization. No process-kill/power-loss durability proof or bound on unresolved/over-budget histories. | | Concurrent stop/publication; route claim and relay lifetime | native-route-stop, shared-https-lifetime, previous-boot-shared-https, relay-lifecycle-barrier, relay-authorization, relay-staging, relay-fence | Fenced route removal, relay owner, ready/removed protocol, authenticated observations | Native DNS/TLS/browser behavior; old-boot uncertainty and external routing. | +| Shared host helper instance: overlapping ups, session adoption, stale create decisions, replacement before finalize, epoch-record failure, interrupted cleanup, concurrent down, displaced launches, failed down session stop | shared-helper-instance | `src/lib/project-shared-helpers.ts` (shutdown generation, `assertCurrent`, `SharedHelperSessionOwnership`); `src/lib/project-lifecycle-controller.ts` session decision, the locked creation transaction (revalidate, publish, create, environment, record, stop on failure, all under one lock), launches fenced by the lock and a token check, abort and signal cleanup through the same epoch decision (`runIfOwned`), an unknown exact-presence answer failing the inspection, and the locked conditional finalize; `stopSessionBeforeReleasingHelpers` (down, restart, native retirement keep holders when the session stop fails); `runMuxCommand` bounds every lifecycle and backend mux call so it settles before the lock is released, and reports a timeout as unknown (never absence); `tests/mux-command-timeout.test.ts`; `tests/project-lifecycle-controller-ownership.test.ts`, `tests/project-shared-helpers.test.ts`, `tests/native-project-down.test.ts`, e2e `shared-host-helpers` | Helper holder/rollback and claim protocol are not modeled (claim fixtures cover them); timeout settlement is an assumption the code does not guarantee (a killed tmux client's queued command may still apply, so a late kill or window command can reach a newer session; consequences and the session-ID fence, not in this slice, are in the README); zellij is refused for shared-helper lifecycle sessions (token written after creation); the gap between a down's session stop and its release (an up that captured after the stopping mark can still succeed there) is a documented first-slice limitation; real mux/process/lock-file behavior. | | Dependency rebind, incomplete initializer, terminal reuse | dependency-rebind, stopped-pool-startup, restore-history | Dependency claim/readiness, saved execution, restore history and source convergence tests | Actual installer failure, cache corruption, retained application state and resource plateau. | | MCP cold start, cancellation and duplicate launches | mcp-startup, mcp-startup-cancellation | Shared MCP bundle identity, startup deduplication and cancellation regressions | Fresh supported client sessions, full workflow coverage and measured overhead (WU16/WU20/WU28). | | Software install/upgrade/rollback, active launcher versus selection, compatibility | **Coverage gap:** no maintained model here is an installer/selection refinement | `scripts/install-prerelease.py`, prerelease-install/Python installer tests, candidate-install contract | Add a bounded selection/lock/crash/compatibility model before primary-driver promotion; real official-artifact upgrade/rollback still required. | @@ -85,6 +86,7 @@ liveness/reachability controls have their own documented assumptions. | [fence-pending](../../tests/models/tla/fence-pending/Fence.tla) | 7 | 2 | | [relay-staging](../../tests/models/tla/relay-staging/Stage.tla) | 9 | 2 | | [relay-fence](../../tests/models/tla/relay-fence/RelayFence.tla) | 16 | 2 | +| [shared-helper-instance](../../tests/models/tla/shared-helper-instance/Instance.tla) | 7,649 | 11 | | [restore-history](../../tests/models/tla/restore-history/RestoreHistory.tla) | See named result contract | 2 | | [graph-admission](../../tests/models/tla/graph-admission/Admission.tla) | See named result contract | 2 | | [balloon-reuse](../../tests/models/tla/balloon-reuse/BalloonReuse.tla) | See named result contract | 2 | diff --git a/scripts/lib/tla-runtime-models.ts b/scripts/lib/tla-runtime-models.ts index fdc631a0c..3da1f022a 100644 --- a/scripts/lib/tla-runtime-models.ts +++ b/scripts/lib/tla-runtime-models.ts @@ -1234,6 +1234,212 @@ const contracts: readonly ModelContract[] = [ action: "Discard", fields: ['phase = "discarded"', "alive = TRUE"], }, + { + // Shared helper instance lifecycle-session ownership (HACK-1142): the + // controller's session decision, the locked creation transaction + // (revalidate, publish, create, environment, record) and its failure + // outcomes and cleanup, fenced process launches, the locked conditional + // finalize, bounded mux calls that settle under the lock (with controls + // for abandoned calls and timeouts read as absence), creator cleanup by + // session epoch, and a down whose session stop fails; see + // tests/models/tla/README.md. + name: "shared-helper-instance", + module: "Instance", + states: 7649, + invariant: "SessionSafe", + action: "Succeed", + fields: [ + 'session = "absent"', + 'epoch = <<"b", "adopted">>', + 'pc = [a |-> "done", b |-> "succeeded"]', + ], + // TLC's two workers may report either up as the adopter, and the kill + // may land after the adopter's Succeed (reported at Cleanup). + alternativeWitnesses: [ + { + action: "Succeed", + fields: [ + 'session = "absent"', + 'epoch = <<"a", "adopted">>', + 'pc = [a |-> "succeeded", b |-> "done"]', + ], + }, + { + action: "Cleanup", + fields: [ + 'session = "absent"', + 'epoch = <<"b", "adopted">>', + 'pc = [a |-> "done", b |-> "succeeded"]', + ], + }, + { + action: "Cleanup", + fields: [ + 'session = "absent"', + 'epoch = <<"a", "adopted">>', + 'pc = [a |-> "succeeded", b |-> "done"]', + ], + }, + ], + additionalControls: [ + { + name: "no-revalidation", + negative: true, + invariant: "Reclaimable", + action: "Publish", + fields: ['session = "alive"', 'token = "a"', 'entry = "b"'], + alternativeWitnesses: [ + { + action: "Publish", + fields: ['session = "alive"', 'token = "b"', 'entry = "a"'], + }, + ], + }, + { + name: "epoch-failure-unstopped", + negative: true, + invariant: "Reclaimable", + action: "RecordFailUnstopped", + fields: ['session = "alive"', 'token = "a"', 'entry = "none"'], + alternativeWitnesses: [ + { + action: "RecordFailUnstopped", + fields: ['session = "alive"', 'token = "b"', 'entry = "none"'], + }, + ], + }, + { + name: "stale-finalize", + negative: true, + invariant: "Reclaimable", + action: "Finalize", + fields: ['session = "alive"', 'token = "b"', 'entry = "a"'], + // TLC's two workers may report either up as the displaced one. + alternativeWitnesses: [ + { + action: "Finalize", + fields: ['session = "alive"', 'token = "a"', 'entry = "b"'], + }, + ], + }, + { + name: "abandoned-create", + negative: true, + invariant: "Reclaimable", + action: "LateCreate", + fields: ['session = "alive"', 'token = "a"', 'entry = "none"'], + alternativeWitnesses: [ + { + action: "LateCreate", + fields: ['session = "alive"', 'token = "b"', 'entry = "none"'], + }, + ], + }, + { + name: "abandoned-replace", + negative: true, + invariant: "SessionSafe", + action: "LateKill", + fields: [ + 'session = "absent"', + 'pc = [a |-> "succeeded", b |-> "done"]', + ], + // The stale kill can land before or after the victim's Succeed; + // either up can be the victim. + alternativeWitnesses: [ + { + action: "LateKill", + fields: [ + 'session = "absent"', + 'pc = [a |-> "done", b |-> "succeeded"]', + ], + }, + { + action: "Succeed", + fields: [ + 'session = "absent"', + 'pc = [a |-> "succeeded", b |-> "done"]', + ], + }, + { + action: "Succeed", + fields: [ + 'session = "absent"', + 'pc = [a |-> "done", b |-> "succeeded"]', + ], + }, + ], + }, + { + name: "unlocked-failure-cleanup", + negative: true, + invariant: "SessionSafe", + action: "LateKill", + fields: [ + 'session = "absent"', + 'pc = [a |-> "done", b |-> "succeeded"]', + ], + // The stale kill can land before or after the victim's Succeed; + // either up can be the victim. + alternativeWitnesses: [ + { + action: "LateKill", + fields: [ + 'session = "absent"', + 'pc = [a |-> "succeeded", b |-> "done"]', + ], + }, + { + action: "Succeed", + fields: [ + 'session = "absent"', + 'pc = [a |-> "done", b |-> "succeeded"]', + ], + }, + { + action: "Succeed", + fields: [ + 'session = "absent"', + 'pc = [a |-> "succeeded", b |-> "done"]', + ], + }, + ], + }, + { + name: "unfenced-launch", + negative: true, + invariant: "NoForeignLaunch", + action: "StartProcesses", + fields: ["foreign = TRUE", 'token = "b"', 'entry = "b"'], + alternativeWitnesses: [ + { + action: "StartProcesses", + fields: ["foreign = TRUE", 'token = "a"', 'entry = "a"'], + }, + ], + }, + { + name: "release-on-failed-stop", + negative: true, + invariant: "HelpersKeptForSession", + action: "DownStopFailed", + fields: ['down = "failed"', "released = TRUE", 'session = "alive"'], + }, + { + name: "timeout-as-absence", + negative: true, + invariant: "Reclaimable", + action: "Publish", + fields: ['session = "alive"', 'token = "a"', 'entry = "b"'], + alternativeWitnesses: [ + { + action: "Publish", + fields: ['session = "alive"', 'token = "b"', 'entry = "a"'], + }, + ], + }, + ], + }, { name: "relay-fence", module: "RelayFence", diff --git a/src/backends/native-project-down.ts b/src/backends/native-project-down.ts index 5b8f4083a..971514e04 100644 --- a/src/backends/native-project-down.ts +++ b/src/backends/native-project-down.ts @@ -87,6 +87,13 @@ export async function nativeProjectDown(opts: { readonly runtime: NativeRuntimeSelection; readonly scope: NativeProjectRunScope; readonly before?: (run: NativeProjectRun) => Promise; + /** + * Runs once the down is committed: after `before` and the ownership and + * finalization rechecks that follow it, immediately before graph cleanup is + * requested. Effects that must only happen for a down that actually stops + * the graph (such as marking shared helpers as stopping) belong here. + */ + readonly beforeCleanup?: (run: NativeProjectRun) => Promise; readonly retireHostProcesses?: (run: NativeProjectRun) => Promise; readonly after?: (run: NativeProjectRun) => Promise; readonly invoke?: typeof invokeNativeRuntime; @@ -176,6 +183,7 @@ export async function nativeProjectDown(opts: { ) { throw finalizationRefused(); } + await opts.beforeCleanup?.(run); await invoke({ runtime: opts.runtime, cwd: opts.scope.projectRoot, diff --git a/src/backends/native-project-lifecycle.ts b/src/backends/native-project-lifecycle.ts index 47f484310..ff83de426 100644 --- a/src/backends/native-project-lifecycle.ts +++ b/src/backends/native-project-lifecycle.ts @@ -4,7 +4,7 @@ */ export function adoptNativeLifecycleCleanup(lifecycle: { readonly signalCleanup: { readonly dispose: () => void } | null; - readonly cleanup: (() => Promise) | null; + readonly cleanup: (() => Promise) | null; }): () => Promise { lifecycle.signalCleanup?.dispose(); return async () => { diff --git a/src/backends/runtime-backend.ts b/src/backends/runtime-backend.ts index 8d586521a..8ab8749c9 100644 --- a/src/backends/runtime-backend.ts +++ b/src/backends/runtime-backend.ts @@ -50,6 +50,13 @@ export interface RuntimeUpOptions extends RuntimeBaseOptions { * non-zero exit (including the 124 deadline) to the phase that produced it. */ readonly onStartupPhase?: (phase: ComposeStartupPhase) => void; + /** + * Awaited right before containers start: after detached image preparation + * (`up --no-start`) and before `up -d`, or before an attached `up`. A throw + * rejects `up` without starting anything, so a caller can refuse to launch + * services whose prerequisites were withdrawn while images were prepared. + */ + readonly beforeStart?: () => Promise; readonly services?: readonly string[]; readonly noDeps?: boolean; readonly forceRecreate?: boolean; @@ -136,6 +143,7 @@ export const composeRuntimeBackend: RuntimeBackend = { return prepared; } opts.onStartupPhase?.("start"); + await opts.beforeStart?.(); return await runPhase( [...buildComposeArgs(opts), "up", "-d", "--no-build", ...scope], startupTimeoutMs @@ -165,6 +173,7 @@ export const composeRuntimeBackend: RuntimeBackend = { }); } + await opts.beforeStart?.(); const cmd = [ ...buildComposeArgs(opts), "up", diff --git a/src/commands/project.ts b/src/commands/project.ts index efd3e65b7..ff71f6a36 100644 --- a/src/commands/project.ts +++ b/src/commands/project.ts @@ -1,11 +1,14 @@ import { createLifecycleProcessController, installLifecycleSignalCleanup, + LIFECYCLE_SESSION_RETAINED, type LifecycleOperationCleanup, + LifecycleSessionRetainedError, resolveLifecycleCommandServiceName, resolveLifecycleCwd, resolveLifecycleSessionName, resolveProjectLifecycleDefinitionHash, + SHARED_HELPER_MUX_UNSUPPORTED_MESSAGE, stopLifecycleProcessController, wrapLifecyclePersistentCommand as wrapSharedLifecyclePersistentCommand, } from "../lib/project-lifecycle-controller.ts"; @@ -288,6 +291,23 @@ import { updateProjectRuntimeServiceEnvFingerprints, upsertProjectRuntimeStateEntry, } from "../lib/project-runtime-state.ts"; +import { + captureSharedHelperShutdownGeneration, + createSharedHelperSessionOwnership, + describeSharedHelperRelease, + describeSharedHelperStart, + isSharedHelperReleaseProblem, + listSharedHelpersHeld, + markSharedHelperInstanceStopping, + type ProcessCleanupOutcome, + releaseProjectSharedHelpers, + type SharedHelperRelease, + type SharedHelperSessionOwnership, + type SharedHelperShutdownGeneration, + SharedHelperSupersededError, + startProjectSharedHelpers, + withTerminationAbort, +} from "../lib/project-shared-helpers.ts"; import { readProjectsRegistry, upsertProjectRegistration, @@ -315,7 +335,11 @@ import { buildWorktreeRetargetWarning, findSameCheckoutRetargetConflicts, } from "../lib/worktree-runtime-target.ts"; -import { resolveMux } from "../mux/mux-resolver.ts"; +import { + type ResolvedMux, + resolveDefaultBackendName, + resolveMux, +} from "../mux/mux-resolver.ts"; import { renderProjectConfigJson, renderProjectEnvConfigYaml, @@ -1838,8 +1862,66 @@ type LifecycleUpResult = { readonly sessionName: string | null; readonly cleanup: LifecycleOperationCleanup | null; readonly signalCleanup: { readonly dispose: () => void } | null; + /** + * Throws {@link SharedHelperSupersededError} once a concurrent `hack down` + * released this instance's shared helpers. Call it right before launching + * services and again after they start. Absent without shared helpers. + */ + readonly assertSharedHelpersCurrent?: (opts?: { + readonly servicesStarted?: boolean; + }) => Promise; + /** + * Call once the instance's containers run (a detached Compose up + * succeeded): later cleanup then stops only the lifecycle processes and + * keeps the shared helper holders for hack down, so a failure after start + * (an up.after hook, a startup check) never stops a helper underneath the + * running containers. Absent without shared helpers. + */ + readonly keepSharedHelpers?: () => void; }; +/** The optional shared helper controls of a lifecycle start, when present. */ +function sharedHelperControls( + start: LifecycleProcessStart +): Pick { + return { + ...(start.assertSharedHelpersCurrent + ? { assertSharedHelpersCurrent: start.assertSharedHelpersCurrent } + : {}), + ...(start.keepSharedHelpers + ? { keepSharedHelpers: start.keepSharedHelpers } + : {}), + }; +} + +/** + * Lifecycle sessions of projects with shared helpers need tmux. zellij + * creates the session first and records its ownership token with a separate + * `rename-tab`, so a start interrupted between the two leaves a live session + * without a token: neither a later up nor hack down can prove ownership, and + * the helper holders kept for it could never be released. tmux sets the + * token as part of `new-session`. Refuses before any hook, helper or process + * runs. Exported for tests. + */ +export async function assertSharedHelperSessionMuxSupported(opts: { + readonly sharedHelpersConfigured: boolean; + readonly needsSession: boolean; + readonly mux: () => Promise; +}): Promise { + if (!(opts.sharedHelpersConfigured && opts.needsSession)) { + return; + } + const mux = await opts.mux(); + const selected = resolveDefaultBackendName(mux); + const backend = selected ? mux.backends.get(selected) : undefined; + // No backend: the controller reports that itself. The controller also + // enforces this at the backend it actually selects, which may differ if + // the mux configuration changes while hooks run. + if (backend && backend.createsWithOwnerToken !== true) { + throw new Error(SHARED_HELPER_MUX_UNSUPPORTED_MESSAGE); + } +} + function hasPersistentLifecycleCommands( commands: readonly ProjectLifecycleCommand[] | undefined ): boolean { @@ -1856,6 +1938,43 @@ async function runLifecycleUpBeforeAndProcesses(opts: { readonly effectiveEnvName: string | null; readonly composeProject: string; }): Promise { + if (opts.cfg.sharedHelpersError) { + // Fail closed before any hook runs: starting without the configured + // helpers would launch services that depend on them. + throw new Error( + `Invalid lifecycle.sharedHelpers; nothing was started: ${opts.cfg.sharedHelpersError}` + ); + } + // Captured before any preparation hook runs: a down that completes while + // the hooks run supersedes this operation (see startProjectSharedHelpers). + const sharedHelperConfigs = opts.cfg.lifecycle?.sharedHelpers ?? []; + await assertSharedHelperSessionMuxSupported({ + sharedHelpersConfigured: sharedHelperConfigs.length > 0, + needsSession: + (opts.cfg.lifecycle?.processes ?? []).length > 0 || + hasPersistentLifecycleCommands(opts.cfg.lifecycle?.up?.before), + mux: () => resolveMux({ project: opts.project }), + }); + const shutdownGeneration = + sharedHelperConfigs.length > 0 + ? await captureSharedHelperShutdownGeneration({ + composeProject: opts.composeProject, + }) + : undefined; + // Records lifecycle-session adoption and creation, so a creator's cleanup + // never kills a session another start adopted (see the ownership type). + const sessionOwnership = + sharedHelperConfigs.length > 0 + ? createSharedHelperSessionOwnership({ + composeProject: opts.composeProject, + // Another start may hold the instance lock while it ensures helpers. + decisionTimeoutMs: + sharedHelperConfigs.reduce( + (sum, helper) => sum + helper.readyTimeoutMs, + 0 + ) + 10_000, + }) + : undefined; const beforeCommands = opts.cfg.lifecycle?.up?.before; const definitionHash = resolveProjectLifecycleDefinitionHash({ beforeCommands, @@ -1880,14 +1999,22 @@ async function runLifecycleUpBeforeAndProcesses(opts: { signalCleanup: null, }; } - const lifecycleStart = await startLifecycleProcesses({ - project: opts.project, - cfg: opts.cfg, - projectName: opts.projectName, - branch: opts.branch, - env: opts.env, - composeProject: opts.composeProject, - definitionHash, + const lifecycleStart = await startWithSharedHelpers({ + ...opts, + shutdownGeneration, + sessionOwnership, + startProcesses: () => + startLifecycleProcesses({ + project: opts.project, + cfg: opts.cfg, + projectName: opts.projectName, + branch: opts.branch, + env: opts.env, + composeProject: opts.composeProject, + definitionHash, + ...(sessionOwnership ? { sessionOwnership } : {}), + }), + abortProcesses: () => Promise.resolve(), }); const hasProcesses = (opts.cfg.lifecycle?.processes ?? []).length > 0; return { @@ -1900,6 +2027,7 @@ async function runLifecycleUpBeforeAndProcesses(opts: { : null, cleanup: lifecycleStart.cleanup, signalCleanup: lifecycleStart.signalCleanup, + ...sharedHelperControls(lifecycleStart), }; } @@ -1911,6 +2039,7 @@ async function runLifecycleUpBeforeAndProcesses(opts: { env: opts.env, composeProject: opts.composeProject, definitionHash, + ...(sessionOwnership ? { sessionOwnership } : {}), }); const beforeCode = await runLifecycleCommands({ title: opts.title, @@ -1936,28 +2065,475 @@ async function runLifecycleUpBeforeAndProcesses(opts: { }; } - try { - await starter.startMany({ - processes: opts.cfg.lifecycle?.processes ?? [], + const started = await startWithSharedHelpers({ + ...opts, + shutdownGeneration, + sessionOwnership, + startProcesses: async () => { + await starter.startMany({ + processes: opts.cfg.lifecycle?.processes ?? [], + }); + await starter.finalize(); + return { + cleanup: starter.getOperationCleanup(), + signalCleanup: starter.getSignalCleanup(), + }; + }, + abortProcesses: () => starter.abort(), + }); + + return { + code: 0, + sessionName: starter.hasStarted() ? starter.sessionName : null, + cleanup: started.cleanup, + signalCleanup: started.signalCleanup, + ...sharedHelperControls(started), + }; +} + +/** + * After lifecycle processes failed to start: abort them, then release the + * shared helper holders this start added, unless the abort kept the session + * (its stop could not be confirmed). Releasing then could stop a helper + * underneath the retained processes, so the holders stay for hack down. + */ +export async function releaseAfterFailedProcessStart(opts: { + readonly error: unknown; + readonly abortProcesses: () => Promise; + readonly rollbackHelpers: (() => Promise) | null; +}): Promise { + const aborted = await opts.abortProcesses(); + const retained = + aborted === LIFECYCLE_SESSION_RETAINED || + opts.error instanceof LifecycleSessionRetainedError; + if (retained) { + logger.warn({ + message: + "Kept this instance's shared helpers with its lifecycle session, which could not be confirmed stopped. Run hack down to stop both.", }); - await starter.finalize(); + return; + } + await opts.rollbackHelpers?.(); +} + +/** + * What happened to the instance's containers before its helpers would be + * released: "stopped" (Compose or graph shutdown succeeded), "unconfirmed" + * (shutdown failed, timed out or was interrupted) or "kept" (deliberately + * left running, as by hack restart, which preserves the Compose runtime). + */ +export type ContainerShutdown = "stopped" | "unconfirmed" | "kept"; + +/** + * The SIGINT/SIGTERM cleanup of an up or restart: once a container start has + * begun, keep the shared helpers before cleaning up (only the lifecycle + * processes stop), because the start may already have started services and + * may still start more. Exported for tests. + */ +export function signalCleanupKeepingStartedHelpers(opts: { + readonly cleanup: LifecycleOperationCleanup | null; + readonly containerStartBegun: () => boolean; + readonly keepSharedHelpers: () => (() => void) | undefined; +}): LifecycleOperationCleanup | null { + const cleanup = opts.cleanup; + if (!cleanup) { + return null; + } + return async () => { + if (opts.containerStartBegun()) { + opts.keepSharedHelpers()?.(); + } + return await cleanup(); + }; +} + +/** + * After a container start that did not succeed (Compose failed, timed out or + * was interrupted, possibly after starting some services): keep the shared + * helper holders unless Compose confirms that no container of the project + * exists in a state that may run. Unknown counts as running. Exported for + * tests. + */ +export async function retainHelpersUnlessContainersAbsent(opts: { + readonly probe: () => Promise<"absent" | "present" | "unknown">; + readonly keepSharedHelpers: (() => void) | undefined; +}): Promise { + if (!opts.keepSharedHelpers) { + return; + } + const containers = await opts.probe().catch(() => "unknown" as const); + if (containers !== "absent") { + opts.keepSharedHelpers(); + } +} + +/** Whether any container of the Compose project exists in a state that may run. */ +async function probeComposeContainers(opts: { + readonly project: Awaited>; + readonly composeProjectName: string | null; + readonly profiles: readonly string[]; + readonly env?: Readonly>; +}): Promise<"absent" | "present" | "unknown"> { + const res = await composeRuntimeBackend.psJson({ + composeFiles: [opts.project.composeFile], + composeProject: opts.composeProjectName, + profiles: opts.profiles, + cwd: dirname(opts.project.composeFile), + all: true, + env: opts.env, + }); + if (res.exitCode !== 0) { + return "unknown"; + } + const stopped = new Set(["exited", "dead"]); + return parseJsonLines(res.stdout).some( + (entry) => !stopped.has((getString(entry, "State") ?? "").toLowerCase()) + ) + ? "present" + : "absent"; +} + +/** + * The restart down phase's container outcome. A preserved Compose runtime + * was never shut down: its containers are kept, so their helpers stay held. + * Exported for tests. + */ +export function restartContainerShutdown(opts: { + readonly preserveComposeRuntime: boolean; + readonly downCode: number; +}): ContainerShutdown { + if (opts.preserveComposeRuntime) { + return "kept"; + } + return opts.downCode === 0 ? "stopped" : "unconfirmed"; +} + +/** + * Stop the lifecycle session, then release this instance's shared helper + * holders only once both the session and the containers are confirmed + * stopped. When the session stop failed or its outcome is unknown, or the + * containers were not confirmed stopped, something may still use the + * helpers: every holder is kept (a later hack down releases them) and + * reported as not released, so down, restart and native retirement fail + * visibly. Containers kept on purpose keep their holders too, but that is + * not a failure: nothing is reported. Never throws. + */ +export async function stopSessionBeforeReleasingHelpers(opts: { + readonly stopSession: () => Promise; + /** Only "stopped" lets the holders be released. */ + readonly containers: ContainerShutdown; + /** Returns the helpers whose release failed; must not throw. */ + readonly releaseHelpers: () => Promise; + /** Helpers the instance may still hold, configured or recorded; must not throw. */ + readonly heldHelperNames: () => Promise; +}): Promise<{ + readonly sessionStopFailed: { readonly error: unknown } | null; + readonly notReleased: readonly string[]; +}> { + try { + await opts.stopSession(); } catch (error: unknown) { - await starter.abort(); - if (error instanceof Error) { - throw error; + return { + sessionStopFailed: { error }, + notReleased: await opts.heldHelperNames(), + }; + } + if (opts.containers === "kept") { + return { sessionStopFailed: null, notReleased: [] }; + } + if (opts.containers === "unconfirmed") { + return { + sessionStopFailed: null, + notReleased: await opts.heldHelperNames(), + }; + } + return { sessionStopFailed: null, notReleased: await opts.releaseHelpers() }; +} + +/** + * Run Compose up guarded by this instance's shared helper check: right + * before containers start (via the backend's `beforeStart`) and again after a + * successful start. A concurrent `hack down` that released the helpers is + * reported as `superseded` instead of a successful start. + */ +async function composeUpGuardedBySharedHelpers(opts: { + readonly assertCurrent: LifecycleUpResult["assertSharedHelpersCurrent"]; + readonly up: (beforeStart?: () => Promise) => Promise; +}): Promise< + | { readonly code: number } + | { readonly superseded: SharedHelperSupersededError } +> { + const assertCurrent = opts.assertCurrent; + if (!assertCurrent) { + return { code: await opts.up() }; + } + try { + const code = await opts.up(() => assertCurrent()); + if (code === 0) { + await assertCurrent({ servicesStarted: true }); + } + return { code }; + } catch (error: unknown) { + if (error instanceof SharedHelperSupersededError) { + return { superseded: error }; + } + throw error; + } +} + +export type LifecycleProcessStart = { + readonly cleanup: LifecycleOperationCleanup | null; + readonly signalCleanup: { readonly dispose: () => void } | null; + readonly assertSharedHelpersCurrent?: (opts?: { + readonly servicesStarted?: boolean; + }) => Promise; + /** See {@link LifecycleUpResult.keepSharedHelpers}. */ + readonly keepSharedHelpers?: () => void; +}; + +/** + * Start shared helpers, then lifecycle processes, as one guarded step. + * + * Without shared helpers this is `startProcesses` with `abortProcesses` on + * failure, unchanged. With them, SIGINT/SIGTERM handlers stay installed from + * the first helper start until the combined cleanup is handed back: a signal + * cancels an in-flight helper start, and once this step settles it runs the + * cleanup for whatever was started (process session and helper holders) and + * is re-raised. No window exists in which a started helper is unguarded. + */ +async function startWithSharedHelpers(opts: { + readonly project: Awaited>; + readonly cfg: Awaited>; + readonly env: Readonly>; + readonly composeProject: string; + readonly startProcesses: () => Promise; + readonly abortProcesses: () => Promise; + readonly shutdownGeneration: SharedHelperShutdownGeneration | undefined; + readonly sessionOwnership: SharedHelperSessionOwnership | undefined; +}): Promise { + const failed = (error: unknown): Error => + error instanceof Error + ? error + : new Error("Failed to start lifecycle processes"); + if ((opts.cfg.lifecycle?.sharedHelpers ?? []).length === 0) { + try { + return await opts.startProcesses(); + } catch (error: unknown) { + await opts.abortProcesses(); + throw failed(error); } - throw new Error("Failed to start lifecycle processes"); } + let onSignalled: () => Promise = () => Promise.resolve(); + return await withTerminationAbort( + (signal) => + runGuardedSharedHelperStart({ + startHelpers: () => startLifecycleSharedHelpers({ ...opts, signal }), + startProcesses: opts.startProcesses, + abortProcesses: opts.abortProcesses, + sessionOwnership: opts.sessionOwnership, + setSignalCleanup: (cleanup) => { + onSignalled = cleanup; + }, + }), + { onSignalled: () => onSignalled() } + ); +} +/** + * The body of {@link startWithSharedHelpers}, run inside its termination + * guard. `setSignalCleanup` names what a caught signal must undo once this + * settles: the helper rollback while processes start, then the combined + * cleanup. A failed process start does its own cleanup here (keeping the + * helpers when the session was retained), so it leaves no signal cleanup + * behind that could release them a second time. Exported for tests. + */ +export async function runGuardedSharedHelperStart(opts: { + readonly startHelpers: () => Promise; + readonly startProcesses: () => Promise; + readonly abortProcesses: () => Promise; + readonly sessionOwnership: SharedHelperSessionOwnership | undefined; + readonly setSignalCleanup: (cleanup: () => Promise) => void; +}): Promise { + let sharedHelpers: LifecycleSharedHelpers | null = null; + let started: LifecycleProcessStart; + try { + sharedHelpers = await opts.startHelpers(); + opts.setSignalCleanup(sharedHelpers.rollback); + started = await opts.startProcesses(); + } catch (error: unknown) { + // This catch owns the cleanup from here on, whatever it decides. + opts.setSignalCleanup(() => Promise.resolve()); + await releaseAfterFailedProcessStart({ + error, + abortProcesses: opts.abortProcesses, + rollbackHelpers: sharedHelpers ? sharedHelpers.rollback : null, + }); + throw error instanceof Error + ? error + : new Error("Failed to start lifecycle processes"); + } + const combined = withSharedHelperCleanup({ + ...started, + sharedHelpers, + sessionOwnership: opts.sessionOwnership, + }); + opts.setSignalCleanup(async () => { + await combined.cleanup?.(); + }); return { - code: 0, - sessionName: starter.hasStarted() ? starter.sessionName : null, - cleanup: starter.getOperationCleanup(), - signalCleanup: starter.getSignalCleanup(), + ...combined, + assertSharedHelpersCurrent: sharedHelpers.assertCurrent, }; } -async function startLifecycleProcesses(opts: { +export type LifecycleSharedHelpers = { + readonly configured: boolean; + readonly rollback: () => Promise; + readonly assertCurrent: (opts?: { + readonly servicesStarted?: boolean; + }) => Promise; +}; + +/** + * Ensure `lifecycle.sharedHelpers` for this instance. A failure, including a + * start cancelled through `signal`, throws after releasing what this run added. + */ +async function startLifecycleSharedHelpers(opts: { + readonly project: Awaited>; + readonly cfg: Awaited>; + readonly env: Readonly>; + readonly composeProject: string; + readonly signal: AbortSignal; + readonly shutdownGeneration: SharedHelperShutdownGeneration | undefined; +}): Promise { + const started = await startProjectSharedHelpers({ + helpers: opts.cfg.lifecycle?.sharedHelpers, + projectRoot: opts.project.projectRoot, + composeProject: opts.composeProject, + env: opts.env, + signal: opts.signal, + ...(opts.shutdownGeneration + ? { shutdownGeneration: opts.shutdownGeneration } + : {}), + report: (record) => { + if (record.result.state !== "failed") { + logger.info({ message: describeSharedHelperStart(record) }); + } + }, + }); + return { + configured: true, + rollback: async () => { + reportSharedHelperReleases(await started.rollback()); + }, + assertCurrent: started.assertCurrent, + }; +} + +/** + * Fold the shared helper rollback into the operation cleanup. The process + * controller's own signal handler only knows its cleanup, so it is replaced: + * returning no signal cleanup makes the caller install one for the combined + * cleanup. + * + * When process cleanup is undecidable (the instance lock was busy), the + * helper rollback is skipped too: keeping the session while releasing its + * helpers would leave a running session whose helpers were stopped. Both are + * kept for `hack down`. Likewise, once `keepSharedHelpers` was called (the + * containers run), the cleanup stops the processes but keeps the helpers. + */ +export function withSharedHelperCleanup(opts: { + readonly cleanup: LifecycleOperationCleanup | null; + readonly signalCleanup: { readonly dispose: () => void } | null; + readonly sharedHelpers: LifecycleSharedHelpers | null; + readonly sessionOwnership: + | Pick + | undefined; +}): { + readonly cleanup: LifecycleOperationCleanup | null; + readonly signalCleanup: { readonly dispose: () => void } | null; + readonly keepSharedHelpers?: () => void; +} { + const sharedHelpers = opts.sharedHelpers; + let containersRunning = false; + const runProcessCleanup = + opts.sessionOwnership?.runIfOwned ?? + (async (cleanup: () => Promise): Promise => { + await cleanup(); + return "ran"; + }); + if (!sharedHelpers?.configured) { + return { cleanup: opts.cleanup, signalCleanup: opts.signalCleanup }; + } + opts.signalCleanup?.dispose(); + return { + keepSharedHelpers: () => { + containersRunning = true; + }, + cleanup: async () => { + const processCleanup = opts.cleanup; + let outcome: ProcessCleanupOutcome | null = null; + // The session could not be confirmed stopped and was kept: its shared + // helpers must stay too, or they would stop underneath it. + let sessionRetained = false; + try { + // Another start of this instance may have adopted the lifecycle + // session this start created; adoption keeps the creator's token, so + // only clean up while the session is still this start's own. + if (processCleanup) { + outcome = await runProcessCleanup(async () => { + if ((await processCleanup()) === LIFECYCLE_SESSION_RETAINED) { + sessionRetained = true; + } + }); + if (outcome === "undecidable") { + logger.warn({ + message: + "Kept this instance's lifecycle session and its shared helpers: the shared helper instance lock was busy, so whether another hack up adopted them could not be checked. Run hack down to stop them if nothing uses them.", + }); + } + } + } finally { + if (sessionRetained) { + logger.warn({ + message: + "Kept this instance's shared helpers with its lifecycle session, which could not be confirmed stopped. Run hack down to stop both.", + }); + } else if (containersRunning) { + logger.warn({ + message: + "Kept this instance's shared helpers: its containers are running. Run hack down to stop both.", + }); + } else if (outcome !== "undecidable") { + await sharedHelpers.rollback(); + } + } + }, + signalCleanup: null, + }; +} + +function reportSharedHelperReleases( + releases: readonly SharedHelperRelease[] +): readonly SharedHelperRelease[] { + for (const release of releases) { + const message = describeSharedHelperRelease(release); + if (isSharedHelperReleaseProblem(release)) { + logger.warn({ message }); + } else { + logger.info({ message }); + } + } + return releases.filter(isSharedHelperReleaseProblem); +} + +/** + * Start the configured lifecycle processes. On failure the started session + * is aborted; when that abort cannot confirm the session stopped, the + * failure is raised as LifecycleSessionRetainedError so the caller keeps the + * shared helpers the session uses. Exported for regression tests. + */ +export async function startLifecycleProcesses(opts: { readonly project: Awaited>; readonly cfg: Awaited>; readonly projectName: string; @@ -1965,6 +2541,9 @@ async function startLifecycleProcesses(opts: { readonly env: Readonly>; readonly composeProject: string; readonly definitionHash: string; + readonly sessionOwnership?: SharedHelperSessionOwnership; + /** Mux backends; defaults to the project's configured backends. */ + readonly mux?: () => Promise; }): Promise<{ readonly cleanup: LifecycleOperationCleanup | null; readonly signalCleanup: { readonly dispose: () => void } | null; @@ -1984,13 +2563,16 @@ async function startLifecycleProcesses(opts: { } const starter = createLifecycleProcessController({ - mux: () => resolveMux({ project: opts.project }), + mux: opts.mux ?? (() => resolveMux({ project: opts.project })), project: opts.project, projectName: opts.projectName, branch: opts.branch, env: opts.env, composeProject: opts.composeProject, definitionHash: opts.definitionHash, + ...(opts.sessionOwnership + ? { sessionOwnership: opts.sessionOwnership } + : {}), }); try { await starter.startMany({ processes }); @@ -2000,7 +2582,9 @@ async function startLifecycleProcesses(opts: { signalCleanup: starter.getSignalCleanup(), }; } catch (error: unknown) { - await starter.abort(); + if ((await starter.abort()) === LIFECYCLE_SESSION_RETAINED) { + throw new LifecycleSessionRetainedError(error); + } if (error instanceof Error) { throw error; } @@ -2044,7 +2628,26 @@ async function stopLifecycleProcesses(opts: { readonly branch: string | null; readonly composeProject: string; }): Promise { - await stopLifecycleProcessController({ + const outcome = await stopSessionBeforeReleasingHelpers({ + stopSession: () => stopLifecycleSession(opts), + // Native retirement runs after the graph's own shutdown succeeded. + containers: "stopped", + releaseHelpers: () => releaseLifecycleSharedHelpersReporting(opts), + heldHelperNames: () => heldSharedHelperNames(opts), + }); + if (outcome.sessionStopFailed) { + warnSharedHelpersKeptForSession(outcome.notReleased); + throw outcome.sessionStopFailed.error; + } + if (outcome.notReleased.length > 0) { + throw new Error(sharedHelperReleaseFailureMessage(outcome.notReleased)); + } +} + +function stopLifecycleSession( + opts: Parameters[0] +): Promise { + return stopLifecycleProcessController({ ...opts, definitionHash: resolveProjectLifecycleDefinitionHash({ beforeCommands: opts.cfg.lifecycle?.up?.before, @@ -2053,18 +2656,115 @@ async function stopLifecycleProcesses(opts: { }); } -async function stopLifecycleProcessesBestEffort( - opts: Parameters[0] -): Promise { +function heldSharedHelperNames(opts: { + readonly cfg: Awaited>; + readonly composeProject: string; +}): Promise { + return listSharedHelpersHeld({ + helpers: opts.cfg.lifecycle?.sharedHelpers, + composeProject: opts.composeProject, + }); +} + +function warnSharedHelpersKeptForSession(kept: readonly string[]): void { + if (kept.length > 0) { + logger.warn({ + message: `Kept shared helpers ${kept.join(", ")}: this instance's lifecycle session could not be confirmed stopped. Run hack down again once it can be stopped.`, + }); + } +} + +/** + * Release this instance's shared helper holders and report each outcome. + * Returns the names of helpers whose release failed or whose stop was not + * confirmed; their claims are kept for the next down. Never throws. + */ +async function releaseLifecycleSharedHelpersReporting(opts: { + readonly cfg: Awaited>; + readonly composeProject: string; +}): Promise { try { - await stopLifecycleProcesses(opts); + return reportSharedHelperReleases( + await releaseProjectSharedHelpers({ + helpers: opts.cfg.lifecycle?.sharedHelpers, + composeProject: opts.composeProject, + }) + ).map((release) => release.name); } catch (error: unknown) { - const message = - error instanceof Error - ? error.message - : "Failed to stop lifecycle processes"; - logger.warn({ message }); + logger.warn({ + message: + error instanceof Error ? error.message : "Shared helper release failed", + }); + return (opts.cfg.lifecycle?.sharedHelpers ?? []).map( + (helper) => helper.name + ); + } +} + +function sharedHelperReleaseFailureMessage(names: readonly string[]): string { + return `Shared helper release incomplete: ${names.join(", ")}. The claim was kept; inspect it and run hack down again.`; +} + +/** + * Mark the instance as stopping before `hack down` stops anything, so a + * concurrent `hack up` of the same instance refuses to launch services. + */ +async function markLifecycleSharedHelpersStopping(opts: { + readonly cfg: Awaited>; + readonly composeProject: string; +}): Promise { + const marked = await markSharedHelperInstanceStopping({ + helpers: opts.cfg.lifecycle?.sharedHelpers, + composeProject: opts.composeProject, + }); + if (!marked) { + logger.warn({ + message: + "Could not mark shared helpers as stopping; a concurrent hack up of this instance is not refused.", + }); + } +} + +/** + * Stop lifecycle processes best-effort (failures are warnings) and release + * shared helpers. Returns the helpers that were not released: unlike process + * cleanup, an unreleased helper keeps running, so callers on the main down + * path must fail the command. When the session stop failed or is unknown, + * every helper is kept with the session and returned. Never throws (safe for + * signal cleanup). + */ +async function stopLifecycleProcessesBestEffort( + opts: Parameters[0] & { + /** Only "stopped" lets the helper holders be released. */ + readonly containers: ContainerShutdown; + } +): Promise { + const outcome = await stopSessionBeforeReleasingHelpers({ + stopSession: () => stopLifecycleSession(opts), + containers: opts.containers, + releaseHelpers: () => releaseLifecycleSharedHelpersReporting(opts), + heldHelperNames: () => heldSharedHelperNames(opts), + }); + if ( + !outcome.sessionStopFailed && + opts.containers === "unconfirmed" && + outcome.notReleased.length > 0 + ) { + logger.warn({ + message: `Kept shared helpers ${outcome.notReleased.join(", ")}: this instance's containers were not confirmed stopped. Run hack down again once they are.`, + }); + } + if (outcome.sessionStopFailed) { + const { error } = outcome.sessionStopFailed; + logger.warn({ + message: + error instanceof Error + ? error.message + : "Failed to stop lifecycle processes", + }); + warnSharedHelpersKeptForSession(outcome.notReleased); } + return outcome.notReleased; } export async function mergeLifecycleCommandEnv( @@ -5107,9 +5807,17 @@ async function handleNativeUp({ message: "Native up lifecycle preparation failed; values omitted.", }); } + // The native backend launches services after this returns: refuse when a + // concurrent down already released this instance's shared helpers. + await assertNativeSharedHelpersCurrent({ lifecycle, cleanup }); return { cleanup, ready: async () => { + await assertNativeSharedHelpersCurrent({ + lifecycle, + cleanup, + servicesStarted: true, + }); const code = await runLifecycleCommands({ title: "Lifecycle (native up after)", commands: cfg.lifecycle?.up?.after, @@ -5429,6 +6137,10 @@ async function runUpCommand({ message: `Failed to parse ${configPath}: ${cfg.parseError}`, }); } + const sharedHelperRefusal = refuseInvalidSharedHelpers({ cfg, json }); + if (sharedHelperRefusal !== null) { + return sharedHelperRefusal; + } const baseProjectName = await resolveComposeProjectName({ project, cfg }); const composeProjectName = branch ? `${baseProjectName}--${branch}` : null; @@ -5503,8 +6215,21 @@ async function runUpCommand({ let lifecycleCleanup: LifecycleOperationCleanup | null = null; let lifecycleSignalCleanup: { readonly dispose: () => void } | null = null; + let assertHelpersCurrent: LifecycleUpResult["assertSharedHelpersCurrent"]; + let keepSharedHelpers: LifecycleUpResult["keepSharedHelpers"]; + const retainHelpersIfContainersMayRun = (): Promise => + retainHelpersUnlessContainersAbsent({ + probe: () => + probeComposeContainers({ + project, + composeProjectName, + profiles, + env: envOverrides.env, + }), + keepSharedHelpers, + }); try { - const lifecycleUp = serviceScoped + const lifecycleUp: LifecycleUpResult = serviceScoped ? { code: 0, cleanup: null, signalCleanup: null, sessionName: null } : await runLifecycleUpBeforeAndProcesses({ title: "Lifecycle (up before)", @@ -5535,6 +6260,8 @@ async function runUpCommand({ } lifecycleCleanup = lifecycleUp.cleanup; lifecycleSignalCleanup = lifecycleUp.signalCleanup; + assertHelpersCurrent = lifecycleUp.assertSharedHelpersCurrent; + keepSharedHelpers = lifecycleUp.keepSharedHelpers; } catch (error: unknown) { const message = error instanceof Error @@ -5550,9 +6277,19 @@ async function runUpCommand({ return 1; } + // Set right before Compose may start containers. A signal from then on + // keeps the shared helpers: a start still in flight can start services + // after any snapshot, so no probe could prove them absent. + let containerStartBegun = false; const signalCleanup = lifecycleSignalCleanup ?? - installLifecycleSignalCleanup({ cleanup: lifecycleCleanup }); + installLifecycleSignalCleanup({ + cleanup: signalCleanupKeepingStartedHelpers({ + cleanup: lifecycleCleanup, + containerStartBegun: () => containerStartBegun, + keepSharedHelpers: () => keepSharedHelpers, + }), + }); try { // Hooks may register or remove runtime aliases; render overrides only after // successful preparation, while lifecycle cleanup remains installed. @@ -5611,6 +6348,10 @@ async function runUpCommand({ env: envOverrides.env, }); if (bootstrapCode !== 0) { + // Release whatever this operation started, like every later failure. + // (Today the cache bootstrap only runs for a service-scoped up, which + // starts no lifecycle work, so this cleanup is null there.) + await lifecycleCleanup?.(); const failure = dependencyCacheBootstrapFailure({ code: bootstrapCode, timeoutMs: detach ? startupTimeoutMs : undefined, @@ -5625,26 +6366,56 @@ async function runUpCommand({ return bootstrapCode; } const upPhase = trackComposeStartupPhase(); - const upCode = await composeRuntimeBackend.up({ - dependencyProgress: resolveDependencyCacheProgress({ - cache: dependencyCache, - project: composeProjectName, - baseProject: baseProjectName, - }), - startupTimeoutMs, - imageTimeoutMs: budgets.imageTimeoutMs, - onStartupPhase: upPhase.onStartupPhase, - composeFiles: composeFilesWithEnv, - composeProject: composeProjectName, - profiles, - detach, - cwd: dirname(project.composeFile), - env: envOverrides.env, - routeStdoutToStderr: json, - services: serviceScoped ? targetServices : undefined, - noDeps: serviceScoped, + containerStartBegun = true; + const guarded = await composeUpGuardedBySharedHelpers({ + assertCurrent: assertHelpersCurrent, + up: (beforeStart) => + composeRuntimeBackend.up({ + ...(beforeStart ? { beforeStart } : {}), + dependencyProgress: resolveDependencyCacheProgress({ + cache: dependencyCache, + project: composeProjectName, + baseProject: baseProjectName, + }), + startupTimeoutMs, + imageTimeoutMs: budgets.imageTimeoutMs, + onStartupPhase: upPhase.onStartupPhase, + composeFiles: composeFilesWithEnv, + composeProject: composeProjectName, + profiles, + detach, + cwd: dirname(project.composeFile), + env: envOverrides.env, + routeStdoutToStderr: json, + services: serviceScoped ? targetServices : undefined, + noDeps: serviceScoped, + }), }); + if ("superseded" in guarded) { + const error = guarded.superseded; + await lifecycleCleanup?.(); + await removeProjectRuntimeStateEntry({ + projectDir: project.projectDir, + composeProject: lifecycleComposeProject, + }); + if (json) { + return emitLifecycleResult({ + result: errorResult({ + code: "E_LIFECYCLE_FAILED", + message: error.message, + detail: { sharedHelpers: "superseded" }, + }), + exitCode: 1, + }); + } + logger.error({ message: error.message }); + return 1; + } + const upCode = guarded.code; if (upCode !== 0) { + // Compose may have started some services before it failed or timed + // out: their helpers stay unless no container may run. + await retainHelpersIfContainersMayRun(); await lifecycleCleanup?.(); await removeProjectRuntimeStateEntry({ projectDir: project.projectDir, @@ -5670,6 +6441,11 @@ async function runUpCommand({ return upCode; } + if (detach) { + // The containers now run: a later failure keeps their helpers. + keepSharedHelpers?.(); + } + const afterCode = serviceScoped ? 0 : await runLifecycleCommands({ @@ -5764,6 +6540,7 @@ async function runUpCommand({ exitCode: 0, }); } catch (error: unknown) { + await retainHelpersIfContainersMayRun(); await lifecycleCleanup?.(); await removeProjectRuntimeStateEntry({ projectDir: project.projectDir, @@ -5965,6 +6742,10 @@ async function handleDown({ }); await hook("before"); }, + // Only a committed down (its before hook succeeded and the ownership + // and finalization rechecks passed) may supersede a concurrent up. + beforeCleanup: () => + markLifecycleSharedHelpersStopping({ cfg, composeProject }), after: async () => { if (hasHooks) { await hook("after"); @@ -6085,6 +6866,10 @@ async function runDownCommand({ message: `Failed to parse ${configPath}: ${cfg.parseError}`, }); } + const sharedHelperRefusal = refuseInvalidSharedHelpers({ cfg, json }); + if (sharedHelperRefusal !== null) { + return sharedHelperRefusal; + } const baseProjectName = await resolveComposeProjectName({ project, cfg }); let runtime: readonly RuntimeProject[] | null = null; @@ -6148,19 +6933,26 @@ async function runDownCommand({ ? await readComposeServiceStates({ project, composeProjectName, profiles }) : []; - const lifecycleDownCleanup = async (): Promise => { + // Holders are released only once Compose confirmed the containers + // stopped; an interrupted or failed shutdown keeps them for the next down. + const lifecycleDownCleanup = async ( + containers: ContainerShutdown + ): Promise => await stopLifecycleProcessesBestEffort({ project, cfg, projectName, branch, composeProject: lifecycleComposeProject, + containers, }); - }; const signalCleanup = installLifecycleSignalCleanup({ - cleanup: lifecycleDownCleanup, + cleanup: async () => { + await lifecycleDownCleanup("unconfirmed"); + }, }); let code: number; + let helperReleaseFailures: readonly string[] = []; try { const beforeCode = await runLifecycleCommands({ title: "Lifecycle (down before)", @@ -6183,6 +6975,10 @@ async function runDownCommand({ return beforeCode; } + await markLifecycleSharedHelpersStopping({ + cfg, + composeProject: lifecycleComposeProject, + }); code = await composeRuntimeBackend.down({ composeFiles: [project.composeFile], composeProject: composeProjectName, @@ -6190,7 +6986,9 @@ async function runDownCommand({ cwd: dirname(project.composeFile), routeStdoutToStderr: json, }); - await lifecycleDownCleanup(); + helperReleaseFailures = await lifecycleDownCleanup( + code === 0 ? "stopped" : "unconfirmed" + ); } finally { signalCleanup.dispose(); } @@ -6208,6 +7006,21 @@ async function runDownCommand({ } return code; } + if (helperReleaseFailures.length > 0) { + const message = sharedHelperReleaseFailureMessage(helperReleaseFailures); + if (json) { + return emitLifecycleResult({ + result: errorResult({ + code: "E_LIFECYCLE_FAILED", + message, + detail: { sharedHelpers: [...helperReleaseFailures] }, + }), + exitCode: 1, + }); + } + logger.error({ message }); + return 1; + } const cacheRemoval = await removeDisposableCacheVolumes({ candidates: cachePruneCandidates, }); @@ -6292,20 +7105,33 @@ async function runRestartDownPhase(opts: { readonly routeStdoutToStderr?: boolean; readonly envForCompose: Readonly>; readonly preserveComposeRuntime?: boolean; -}): Promise { - const lifecycleDownCleanup = async (): Promise => { +}): Promise<{ + readonly code: number; + /** Helpers whose release failed; the restart must stop. */ + readonly sharedHelperFailures?: readonly string[]; +}> { + // As in hack down: holders are released only once the containers are + // confirmed stopped. A restart that preserves the Compose runtime keeps + // the containers running, so it keeps their holders (not a failure); the + // up phase then reuses the helpers. + const lifecycleDownCleanup = async ( + containers: ContainerShutdown + ): Promise => await stopLifecycleProcessesBestEffort({ project: opts.project, cfg: opts.cfg, projectName: opts.projectName, branch: opts.branch, composeProject: opts.lifecycleComposeProject, + containers, }); - }; const signalCleanup = installLifecycleSignalCleanup({ - cleanup: lifecycleDownCleanup, + cleanup: async () => { + await lifecycleDownCleanup("unconfirmed"); + }, }); let downCode: number; + let helperReleaseFailures: readonly string[] = []; try { const downBefore = await runLifecycleCommands({ title: "Lifecycle (restart down before)", @@ -6316,9 +7142,13 @@ async function runRestartDownPhase(opts: { composeProject: opts.lifecycleComposeProject, }); if (downBefore !== 0) { - return downBefore; + return { code: downBefore }; } + await markLifecycleSharedHelpersStopping({ + cfg: opts.cfg, + composeProject: opts.lifecycleComposeProject, + }); downCode = opts.preserveComposeRuntime ? 0 : await composeRuntimeBackend.down({ @@ -6328,12 +7158,25 @@ async function runRestartDownPhase(opts: { cwd: dirname(opts.project.composeFile), routeStdoutToStderr: opts.routeStdoutToStderr === true, }); - await lifecycleDownCleanup(); + helperReleaseFailures = await lifecycleDownCleanup( + restartContainerShutdown({ + preserveComposeRuntime: opts.preserveComposeRuntime === true, + downCode, + }) + ); } finally { signalCleanup.dispose(); } if (downCode !== 0) { - return downCode; + return { code: downCode }; + } + if (helperReleaseFailures.length > 0) { + // An unreleased helper keeps running with its claim kept: restarting on + // top of it would hide that, so the restart stops here. + logger.error({ + message: sharedHelperReleaseFailureMessage(helperReleaseFailures), + }); + return { code: 1, sharedHelperFailures: helperReleaseFailures }; } if (!opts.preserveComposeRuntime) { @@ -6351,7 +7194,142 @@ async function runRestartDownPhase(opts: { projectDir: opts.project.projectDir, composeProject: opts.lifecycleComposeProject, }); - return downAfter; + return { code: downAfter }; +} + +/** Native-path shared helper check: cleans up, then fails with a lifecycle error. */ +async function assertNativeSharedHelpersCurrent(opts: { + readonly lifecycle: LifecycleUpResult; + readonly cleanup: () => Promise; + readonly servicesStarted?: boolean; +}): Promise { + try { + await opts.lifecycle.assertSharedHelpersCurrent?.({ + servicesStarted: opts.servicesStarted === true, + }); + } catch (error: unknown) { + if (!(error instanceof SharedHelperSupersededError)) { + throw error; + } + await opts.cleanup(); + throw new HackCliError({ + code: "E_LIFECYCLE_FAILED", + message: error.message, + }); + } +} + +/** + * After a failed restart start, attempt one repair start so the runtime is + * left startable. The repair launches containers, so it runs behind the same + * shared helper guard as the first attempt. + */ +export async function repairFailedRestart(opts: { + readonly budgets: ComposeStartupBudgets; + readonly upCode: number; + readonly composePhase: ComposeStartupPhase; + readonly assertCurrent: LifecycleUpResult["assertSharedHelpersCurrent"]; + readonly cleanup: LifecycleOperationCleanup | null; + /** + * Runs before the cleanup of a failed repair: keeps the shared helpers + * when the failed starts may have left containers running. + */ + readonly retainHelpersIfContainersMayRun?: () => Promise; + readonly projectDir: string; + readonly lifecycleComposeProject: string; + readonly up: (beforeStart?: () => Promise) => Promise; +}): Promise<{ + readonly code: number; + readonly phase: "compose" | "lifecycle"; + readonly composePhase?: ComposeStartupPhase; +}> { + reportComposeStartupBudget({ + budgets: opts.budgets, + detach: true, + repair: true, + }); + const repair = await composeUpGuardedBySharedHelpers({ + assertCurrent: opts.assertCurrent, + up: opts.up, + }); + if ("superseded" in repair) { + return await abandonSupersededRestart({ + error: repair.superseded, + cleanup: opts.cleanup, + projectDir: opts.projectDir, + composeProject: opts.lifecycleComposeProject, + }); + } + if (repair.code === 0) { + logger.warn({ + message: + "Restart failed, but Hack repaired the Compose runtime to a startable state.", + }); + } else { + logger.error({ + message: `Restart failed and automatic runtime repair also failed (exit ${repair.code}). Run 'hack doctor --fix'.`, + }); + await opts.retainHelpersIfContainersMayRun?.(); + await opts.cleanup?.(); + await removeProjectRuntimeStateEntry({ + projectDir: opts.projectDir, + composeProject: opts.lifecycleComposeProject, + }); + } + return { + code: opts.upCode, + phase: "compose", + composePhase: opts.composePhase, + }; +} + +/** + * Refuse a command when `lifecycle.sharedHelpers` is invalid, before any + * lifecycle or runtime effect. Other config parse errors only warn on the + * Compose path, but running without the helpers would launch services that + * need them (or leave a down unable to name what to release). Returns the + * exit code to return, or null to continue. + */ +function refuseInvalidSharedHelpers(opts: { + readonly cfg: Awaited>; + readonly json: boolean; +}): number | null { + if (!opts.cfg.sharedHelpersError) { + return null; + } + const message = `Invalid lifecycle.sharedHelpers; nothing was changed: ${opts.cfg.sharedHelpersError}`; + if (opts.json) { + return emitLifecycleResult({ + result: errorResult({ code: "E_LIFECYCLE_FAILED", message }), + exitCode: 1, + }); + } + logger.error({ message }); + return 1; +} + +/** Restart's response to a concurrent down: clean up this start and stop. */ +async function abandonSupersededRestart(opts: { + readonly error: SharedHelperSupersededError; + readonly cleanup: LifecycleOperationCleanup | null; + readonly projectDir: string; + readonly composeProject: string; +}): Promise<{ readonly code: number; readonly phase: "lifecycle" }> { + await opts.cleanup?.(); + await removeProjectRuntimeStateEntry({ + projectDir: opts.projectDir, + composeProject: opts.composeProject, + }); + logger.error({ message: opts.error.message }); + return { code: 1, phase: "lifecycle" }; +} + +function reportLifecycleSession(sessionName: string | null): void { + if (sessionName) { + logger.info({ + message: `Lifecycle processes running in session: ${sessionName}`, + }); + } } async function runRestartUpPhase(opts: { @@ -6407,6 +7385,19 @@ async function runRestartUpPhase(opts: { let lifecycleCleanup: LifecycleOperationCleanup | null = null; let lifecycleSignalCleanup: { readonly dispose: () => void } | null = null; + let assertHelpersCurrent: LifecycleUpResult["assertSharedHelpersCurrent"]; + let keepSharedHelpers: LifecycleUpResult["keepSharedHelpers"]; + const retainHelpersIfContainersMayRun = (): Promise => + retainHelpersUnlessContainersAbsent({ + probe: () => + probeComposeContainers({ + project: opts.project, + composeProjectName: opts.composeProjectName, + profiles: opts.profiles, + env: envOverrides.env, + }), + keepSharedHelpers, + }); try { const lifecycleUp = await runLifecycleUpBeforeAndProcesses({ title: "Lifecycle (restart up before)", @@ -6421,13 +7412,11 @@ async function runRestartUpPhase(opts: { if (lifecycleUp.code !== 0) { return { code: lifecycleUp.code, phase: "lifecycle" }; } - if (lifecycleUp.sessionName) { - logger.info({ - message: `Lifecycle processes running in session: ${lifecycleUp.sessionName}`, - }); - } + reportLifecycleSession(lifecycleUp.sessionName); lifecycleCleanup = lifecycleUp.cleanup; lifecycleSignalCleanup = lifecycleUp.signalCleanup; + assertHelpersCurrent = lifecycleUp.assertSharedHelpersCurrent; + keepSharedHelpers = lifecycleUp.keepSharedHelpers; } catch (error: unknown) { const message = error instanceof Error @@ -6437,9 +7426,19 @@ async function runRestartUpPhase(opts: { return { code: 1, phase: "lifecycle" }; } + // Set right before Compose may start containers. A signal from then on + // keeps the shared helpers: a start still in flight can start services + // after any snapshot, so no probe could prove them absent. + let containerStartBegun = false; const signalCleanup = lifecycleSignalCleanup ?? - installLifecycleSignalCleanup({ cleanup: lifecycleCleanup }); + installLifecycleSignalCleanup({ + cleanup: signalCleanupKeepingStartedHelpers({ + cleanup: lifecycleCleanup, + containerStartBegun: () => containerStartBegun, + keepSharedHelpers: () => keepSharedHelpers, + }), + }); try { // Hooks may register or remove runtime aliases; render overrides only after // successful preparation, while lifecycle cleanup remains installed. @@ -6490,66 +7489,74 @@ async function runRestartUpPhase(opts: { detach: opts.detach === true, }); const upPhase = trackComposeStartupPhase(); - const upCode = await composeRuntimeBackend.up({ - dependencyProgress: resolveDependencyCacheProgress({ - cache: dependencyCache, - project: opts.composeProjectName, - baseProject: opts.baseProjectName, - }), - startupTimeoutMs: opts.budgets.startupTimeoutMs, - imageTimeoutMs: opts.budgets.imageTimeoutMs, - onStartupPhase: upPhase.onStartupPhase, - composeFiles: composeFilesWithEnv, - composeProject: opts.composeProjectName, - profiles: opts.profiles, - detach: opts.detach === true, - cwd: dirname(opts.project.composeFile), - env: envOverrides.env, - routeStdoutToStderr: opts.routeStdoutToStderr === true, - forceRecreate: true, + containerStartBegun = true; + const guarded = await composeUpGuardedBySharedHelpers({ + assertCurrent: assertHelpersCurrent, + up: (beforeStart) => + composeRuntimeBackend.up({ + ...(beforeStart ? { beforeStart } : {}), + dependencyProgress: resolveDependencyCacheProgress({ + cache: dependencyCache, + project: opts.composeProjectName, + baseProject: opts.baseProjectName, + }), + startupTimeoutMs: opts.budgets.startupTimeoutMs, + imageTimeoutMs: opts.budgets.imageTimeoutMs, + onStartupPhase: upPhase.onStartupPhase, + composeFiles: composeFilesWithEnv, + composeProject: opts.composeProjectName, + profiles: opts.profiles, + detach: opts.detach === true, + cwd: dirname(opts.project.composeFile), + env: envOverrides.env, + routeStdoutToStderr: opts.routeStdoutToStderr === true, + forceRecreate: true, + }), }); + if ("superseded" in guarded) { + return await abandonSupersededRestart({ + error: guarded.superseded, + cleanup: lifecycleCleanup, + projectDir: opts.project.projectDir, + composeProject: opts.lifecycleComposeProject, + }); + } + const upCode = guarded.code; if (upCode !== 0) { - reportComposeStartupBudget({ + return await repairFailedRestart({ budgets: opts.budgets, - detach: true, - repair: true, - }); - const repairCode = await composeRuntimeBackend.up({ - dependencyProgress: resolveDependencyCacheProgress({ - cache: dependencyCache, - project: opts.composeProjectName, - baseProject: opts.baseProjectName, - }), - startupTimeoutMs: opts.budgets.startupTimeoutMs, - imageTimeoutMs: opts.budgets.imageTimeoutMs, - composeFiles: composeFilesWithEnv, - composeProject: opts.composeProjectName, - profiles: opts.profiles, - detach: true, - cwd: dirname(opts.project.composeFile), - env: envOverrides.env, - routeStdoutToStderr: opts.routeStdoutToStderr === true, + upCode, + composePhase: upPhase.phase(), + assertCurrent: assertHelpersCurrent, + cleanup: lifecycleCleanup, + retainHelpersIfContainersMayRun, + projectDir: opts.project.projectDir, + lifecycleComposeProject: opts.lifecycleComposeProject, + up: (beforeStart) => + composeRuntimeBackend.up({ + ...(beforeStart ? { beforeStart } : {}), + dependencyProgress: resolveDependencyCacheProgress({ + cache: dependencyCache, + project: opts.composeProjectName, + baseProject: opts.baseProjectName, + }), + startupTimeoutMs: opts.budgets.startupTimeoutMs, + imageTimeoutMs: opts.budgets.imageTimeoutMs, + composeFiles: composeFilesWithEnv, + composeProject: opts.composeProjectName, + profiles: opts.profiles, + detach: true, + cwd: dirname(opts.project.composeFile), + env: envOverrides.env, + routeStdoutToStderr: opts.routeStdoutToStderr === true, + }), }); - if (repairCode === 0) { - logger.warn({ - message: - "Restart failed, but Hack repaired the Compose runtime to a startable state.", - }); - } else { - logger.error({ - message: `Restart failed and automatic runtime repair also failed (exit ${repairCode}). Run 'hack doctor --fix'.`, - }); - } - if (repairCode !== 0) { - await lifecycleCleanup?.(); - await removeProjectRuntimeStateEntry({ - projectDir: opts.project.projectDir, - composeProject: opts.lifecycleComposeProject, - }); - } - return { code: upCode, phase: "compose", composePhase: upPhase.phase() }; } + if (opts.detach === true) { + // The containers now run: a later failure keeps their helpers. + keepSharedHelpers?.(); + } const upAfter = await runLifecycleCommands({ title: "Lifecycle (restart up after)", commands: opts.cfg.lifecycle?.up?.after, @@ -6563,6 +7570,7 @@ async function runRestartUpPhase(opts: { } return { code: upAfter, phase: "lifecycle" }; } catch (error: unknown) { + await retainHelpersIfContainersMayRun(); await lifecycleCleanup?.(); await removeProjectRuntimeStateEntry({ projectDir: opts.project.projectDir, @@ -6960,6 +7968,10 @@ async function runRestartCommand({ message: `Failed to parse ${configPath}: ${cfg.parseError}`, }); } + const sharedHelperRefusal = refuseInvalidSharedHelpers({ cfg, json }); + if (sharedHelperRefusal !== null) { + return sharedHelperRefusal; + } const baseProjectName = await resolveComposeProjectName({ project, cfg }); const { composeProjectName, lifecycleComposeProject } = resolveRestartTarget({ @@ -7042,7 +8054,7 @@ async function runRestartCommand({ envByService: preflightEnv.preflightEnvByService, }); - const downCode = await runRestartDownPhase({ + const downPhase = await runRestartDownPhase({ project, cfg, projectName, @@ -7054,6 +8066,25 @@ async function runRestartCommand({ routeStdoutToStderr: json, preserveComposeRuntime: true, }); + const downCode = downPhase.code; + if (downPhase.sharedHelperFailures) { + if (json) { + return emitLifecycleResult({ + result: errorResult({ + code: "E_LIFECYCLE_FAILED", + message: sharedHelperReleaseFailureMessage( + downPhase.sharedHelperFailures + ), + detail: { + sharedHelpers: [...downPhase.sharedHelperFailures], + phase: "down", + }, + }), + exitCode: 1, + }); + } + return 1; + } if (downCode !== 0) { if (json) { return emitLifecycleResult({ @@ -7141,12 +8172,14 @@ async function runRestartCommand({ successfulCompletionServices, }); if (startupSnapshotIsIncomplete({ states, failed: startup.failed })) { + // Some containers may still run: keep the helper holders for hack down. await stopLifecycleProcessesBestEffort({ project, cfg, projectName, branch, composeProject: lifecycleComposeProject, + containers: "unconfirmed", }); await removeProjectRuntimeStateEntry({ projectDir: project.projectDir, diff --git a/src/lib/project-lifecycle-controller.ts b/src/lib/project-lifecycle-controller.ts index a62a8e368..cfe55bddd 100644 --- a/src/lib/project-lifecycle-controller.ts +++ b/src/lib/project-lifecycle-controller.ts @@ -1,6 +1,11 @@ import { rm } from "node:fs/promises"; import { dirname, resolve } from "node:path"; -import type { MuxBackend, MuxBackendName } from "../mux/mux-backend.ts"; +import { + type MuxBackend, + type MuxBackendName, + muxCommandTimeoutMs, + runMuxCommand, +} from "../mux/mux-backend.ts"; import { getMuxBackends, type ResolvedMux, @@ -41,6 +46,7 @@ import { inspectListeningTcpPorts, resolveLifecycleSingletonDecision, } from "./project-lifecycle-singleton.ts"; +import type { SharedHelperSessionOwnership } from "./project-shared-helpers.ts"; import { exec } from "./shell.ts"; export type LifecycleControllerProject = { readonly projectRoot: string; @@ -53,7 +59,17 @@ type StartedLifecycleProcess = { readonly panePid?: number; readonly processGroupId?: number; }; -export type LifecycleOperationCleanup = () => Promise; +/** + * Cleanup for what an operation started. Resolves + * {@link LIFECYCLE_SESSION_RETAINED} when a + * session it created could not be confirmed stopped and was kept, with its + * ownership record, for hack down; callers must then keep anything that + * session depends on (such as shared helper holders). + */ +export type LifecycleOperationCleanup = () => Promise; + +/** The value an operation cleanup resolves to when it kept the session. */ +export const LIFECYCLE_SESSION_RETAINED = "session-retained"; const LIFECYCLE_COMMAND_PID_WAIT_ATTEMPTS = 50; const LIFECYCLE_COMMAND_PID_WAIT_INTERVAL_MS = 20; export function resolveLifecycleSessionName(opts: { @@ -173,6 +189,20 @@ async function replaceInspectedLifecycleSession(opts: { `Failed to replace stale lifecycle session: ${opts.sessionName}` ); } + // Retire only the entry that was inspected (by its ownership token), never + // whatever entry is current: another start may have published since. + const inspected = + opts.inspection.decision.kind === "replace" + ? opts.inspection.decision.entry + : null; + if (inspected?.ownershipToken) { + await removeLifecycleStateEntryIfOwned({ + projectDir: opts.projectDir, + composeProject: opts.composeProject, + ownershipToken: inspected.ownershipToken, + }); + return; + } await removeLifecycleStateEntry({ projectDir: opts.projectDir, composeProject: opts.composeProject, @@ -207,6 +237,23 @@ export function createLifecycleProcessController(opts: { readonly cleanup: LifecycleOperationCleanup; }) => { readonly dispose: () => void }; readonly assertFresh?: () => Promise; + /** + * Shared helper session ownership (projects with `sharedHelpers`). The + * session decision (inspect, adopt or replace) and session creation run + * under its lock and are recorded, so a creator's cleanup can tell whether + * another start adopted the session; see SharedHelperSessionOwnership. + */ + readonly sessionOwnership?: Pick< + SharedHelperSessionOwnership, + "withLock" | "record" | "runIfOwned" + >; + /** + * Bound for this controller's own tmux/zellij commands (environment setup, + * window creation, process start). Backend calls use muxCommandTimeoutMs(). + * A timed-out command's client is killed, so the caller stops waiting; the + * server may still apply the command later (see runMuxCommand). + */ + readonly muxCommandTimeoutMs?: number; }): { readonly sessionName: string; startFromCommand: (opts: { @@ -217,7 +264,8 @@ export function createLifecycleProcessController(opts: { readonly processes: readonly ProjectLifecycleProcess[]; }) => Promise; finalize: () => Promise; - abort: () => Promise; + /** Resolves LIFECYCLE_SESSION_RETAINED when the session was kept. */ + abort: () => Promise; hasStarted: () => boolean; getOperationCleanup: () => LifecycleOperationCleanup | null; getSignalCleanup: () => { readonly dispose: () => void } | null; @@ -235,11 +283,25 @@ export function createLifecycleProcessController(opts: { let operationSignalCleanup: { readonly dispose: () => void } | null = null; let sessionCreationSettled: Promise = Promise.resolve(); let nextIndex = 0; + const muxTimeoutMs = opts.muxCommandTimeoutMs ?? muxCommandTimeoutMs(); const resolveSessionDisposition = async (): Promise => { if (sessionDispositionResolved) { return; } + const ownership = opts.sessionOwnership; + if (!ownership) { + await decideSessionDisposition(); + return; + } + await ownership.withLock(async () => { + if ((await decideSessionDisposition()) === "adopted") { + await ownership.record("adopted"); + } + }); + }; + + const decideSessionDisposition = async (): Promise<"adopted" | "create"> => { const mux = await opts.mux(); const resolvedBackend = resolveDefaultBackendName({ mode: mux.mode, @@ -257,6 +319,11 @@ export function createLifecycleProcessController(opts: { if (!backend?.available) { throw new Error(`${resolvedBackend} is not available`); } + // Enforced at the backend actually selected, not only at preflight: the + // mux selection is resolved again here and may have changed since. + if (opts.sessionOwnership && backend.createsWithOwnerToken !== true) { + throw new Error(SHARED_HELPER_MUX_UNSUPPORTED_MESSAGE); + } const entries = await readLifecycleState({ projectDir: opts.project.projectDir, }); @@ -290,7 +357,7 @@ export function createLifecycleProcessController(opts: { startedProcesses.push(...inspection.decision.entry.processes); sessionReady = true; sessionDispositionResolved = true; - return; + return "adopted"; } if (inspection.decision.kind === "replace") { await opts.assertFresh?.(); @@ -303,6 +370,7 @@ export function createLifecycleProcessController(opts: { }); } sessionDispositionResolved = true; + return "create"; }; const ensureSession = async (): Promise => { @@ -337,8 +405,18 @@ export function createLifecycleProcessController(opts: { processes: [], updatedAt: new Date().toISOString(), }; - createdOwnershipToken = ownershipToken; - try { + // true: created; false: definitely not created; "unknown": the create + // call did not settle within its deadline or failed without a result. + let sessionCreated: boolean | "unknown" = false; + // Revalidate, publish pending ownership, create and record as one + // transition. With shared helper ownership it runs under the instance + // lock, so an overlapping start cannot create the session (and finalize + // its entry) between this start's decision and its publication. + const createOwnedSession = async (): Promise => { + if (opts.sessionOwnership) { + await assertSessionStillAbsent(backend); + } + createdOwnershipToken = ownershipToken; // Persist ownership before creating the mux session. If the CLI is // SIGKILLed after session creation, the next up can still prove that // the empty session belongs to this Compose instance and replace it. @@ -347,40 +425,255 @@ export function createLifecycleProcessController(opts: { entry: pendingEntry, }); await opts.assertFresh?.(); + sessionCreated = "unknown"; + // Bounded by the backend: the call returns (a timed-out client is + // killed) before this transaction releases the lock, and a timeout is + // an unknown outcome, so the entry and token are kept for reclaim. The + // server may still create the session after the lock is released; that + // residual is documented in tests/models/tla/README.md (HACK-1849). const created = await backend.createSession({ name: sessionName, cwd: opts.project.projectRoot, lifecycleOwnerToken: ownershipToken, }); if (!created.ok) { + // A failed (possibly killed) create may still have created the + // session: the ownership-token check decides, not this result. throw new Error( describeSessionCreateFailure({ sessionName, stderr: created.stderr }) ); } - if (backendName === "tmux") { - for (const [key, value] of Object.entries(opts.env)) { - await exec( - ["tmux", "set-environment", "-t", sessionName, key, value], - { - stdin: "ignore", - } - ); + sessionCreated = true; + await opts.sessionOwnership?.record("created"); + }; + // Environment setup and failed-creation cleanup belong to the same + // locked transition: a replacement (also under the lock) can neither + // receive this start's environment nor have its session killed by this + // start's name-based cleanup between the token read and the kill. + const createAndPrepareSession = async (): Promise => { + try { + await createOwnedSession(); + if (backendName === "tmux") { + await applyTmuxSessionEnvironment({ + sessionName, + env: opts.env, + timeoutMs: muxTimeoutMs, + }); + } + } catch (error: unknown) { + if (createdOwnershipToken === ownershipToken) { + await releaseFailedSessionCreation({ + backend, + ownershipToken, + sessionCreated, + }); } + throw error; } - } catch (error: unknown) { - await removeLifecycleStateEntryIfOwned({ - projectDir: opts.project.projectDir, - composeProject: opts.composeProject, - ownershipToken, - }); - createdOwnershipToken = null; - throw error; + }; + try { + await underInstanceLock(createAndPrepareSession); } finally { settleSessionCreation(); } sessionReady = true; }; + /** Run under the shared helper instance lock when this start has one. */ + const underInstanceLock = async (run: () => Promise): Promise => + opts.sessionOwnership + ? await opts.sessionOwnership.withLock(run) + : await run(); + + /** + * Whether the persisted entry and the live session still carry this + * start's token (created or adopted). Call under the instance lock. False + * when another start replaced the session or the live token is unreadable. + */ + const sessionStillOwned = async (): Promise => { + const entries = await readLifecycleState({ + projectDir: opts.project.projectDir, + }); + const entry = + entries.find( + (candidate) => candidate.composeProject === opts.composeProject + ) ?? null; + const backend = backendName + ? (await opts.mux()).backends.get(backendName) + : undefined; + const expected = + createdOwnershipToken ?? adoptedEntry?.ownershipToken ?? null; + const persistedOurs = + entry !== null && (entry.ownershipToken ?? null) === expected; + const liveOurs = + expected === null || !backend?.readLifecycleOwnerToken + ? true + : (await backend + .readLifecycleOwnerToken({ name: sessionName }) + .catch(() => undefined)) === expected; + return persistedOurs && liveOurs; + }; + + /** Give up cleanup and finalize authority over a replaced session. */ + const relinquishDisplacedSession = (): never => { + createdOwnershipToken = null; + adoptedEntry = null; + throw new LifecycleSessionDisplacedError(sessionName); + }; + + /** + * Refuse to create when the session is no longer absent: an overlapping + * start created it (and may have finalized its entry) after this start + * decided to create. Nothing is published or changed on refusal. + */ + const assertSessionStillAbsent = async ( + backend: MuxBackend + ): Promise => { + const entries = await readLifecycleState({ + projectDir: opts.project.projectDir, + }); + const entry = + entries.find( + (candidate) => candidate.composeProject === opts.composeProject + ) ?? null; + const inspection = await inspectLifecycleSession({ + backend, + entry, + expectedSessionName: sessionName, + expectedProjectRoot: opts.project.projectRoot, + expectedDefinitionHash: opts.definitionHash, + }); + if (inspection.decision.kind !== "create") { + throw new Error( + `Another hack up of this instance changed lifecycle session ${sessionName} while this one prepared it; nothing was changed. Run the command again.` + ); + } + }; + + /** + * After a failed creation transition. A session that was (or may have + * been) created with this start's token is stopped with that token before + * its entry is removed. Whenever the outcome stays unknown (a mux call hung, + * the stop is unconfirmed), the entry and token are kept so abort(), a + * later up or hack down can still prove ownership and reclaim the session. + * Nothing is left running without an ownership record. + */ + const releaseFailedSessionCreation = async (release: { + readonly backend: MuxBackend; + readonly ownershipToken: string; + readonly sessionCreated: boolean | "unknown"; + }): Promise => { + if (release.sessionCreated !== false) { + const observed = await Promise.resolve( + release.backend.readLifecycleOwnerToken?.({ name: sessionName }) ?? null + ).catch(() => undefined); + if (observed === undefined) { + return; + } + const foreign = observed !== null && observed !== release.ownershipToken; + if (!foreign) { + const stopped = await killLifecycleSessionWithOwnership({ + backend: release.backend, + sessionName, + ownershipToken: release.ownershipToken, + }).catch(() => false); + if (!stopped) { + return; + } + } + } + await removeLifecycleStateEntryIfOwned({ + projectDir: opts.project.projectDir, + composeProject: opts.composeProject, + ownershipToken: release.ownershipToken, + }); + createdOwnershipToken = null; + }; + + const publishFinalEntry = async (): Promise => { + if (startedProcesses.length === 0 || !backendName) { + await removeLifecycleStateEntry({ + projectDir: opts.project.projectDir, + composeProject: opts.composeProject, + }); + return; + } + const ownershipToken = + createdOwnershipToken ?? adoptedEntry?.ownershipToken; + await upsertLifecycleStateEntry({ + projectDir: opts.project.projectDir, + entry: { + composeProject: opts.composeProject, + projectName: opts.projectName, + branch: opts.branch, + sessionName, + backend: backendName, + ...(ownershipToken ? { ownershipToken } : {}), + definitionHash: opts.definitionHash, + processes: startedProcesses, + updatedAt: new Date().toISOString(), + }, + }); + }; + + /** + * Publish the final entry only while the persisted entry and the live + * session still carry this start's token (created or adopted). Otherwise + * another start replaced the session: this start gives up its cleanup and + * finalize authority and fails, leaving the replacement's entry intact. + * With no processes of its own, it removes only an entry it owns or one + * whose session is gone, never another start's live one. + */ + const publishFinalEntryIfStillOwned = async (): Promise => { + const entries = await readLifecycleState({ + projectDir: opts.project.projectDir, + }); + const entry = + entries.find( + (candidate) => candidate.composeProject === opts.composeProject + ) ?? null; + const backend = backendName + ? (await opts.mux()).backends.get(backendName) + : undefined; + if (startedProcesses.length === 0 || !backendName) { + if (entry && (await entryRemovable({ entry, backend }))) { + await removeLifecycleStateEntryIfOwned({ + projectDir: opts.project.projectDir, + composeProject: opts.composeProject, + ownershipToken: entry.ownershipToken ?? "", + }); + } + return; + } + if (!(await sessionStillOwned())) { + relinquishDisplacedSession(); + } + await publishFinalEntry(); + }; + + const entryRemovable = async (check: { + readonly entry: LifecycleStateEntry; + readonly backend: MuxBackend | undefined; + }): Promise => { + if ( + createdOwnershipToken !== null && + check.entry.ownershipToken === createdOwnershipToken + ) { + return true; + } + if (!check.backend?.available) { + return false; + } + const inspection = await inspectLifecycleSession({ + backend: check.backend, + entry: check.entry, + expectedSessionName: sessionName, + expectedProjectRoot: opts.project.projectRoot, + expectedDefinitionHash: opts.definitionHash, + }).catch(() => null); + return inspection?.classification === "absent"; + }; + const startProcess = async ( process: ProjectLifecycleProcess ): Promise => { @@ -406,16 +699,25 @@ export function createLifecycleProcessController(opts: { const launch = await opts.prepareLaunch?.(process); await ensureSession(); await opts.assertFresh?.(); - const started = await startLifecycleProcess({ - backend: backendName as MuxBackendName, - sessionName, - projectRoot: opts.project.projectRoot, - env: opts.env, - index: nextIndex, - process: launch ? { ...process, command: launch.command } : process, - projectDir: opts.project.projectDir, - composeProject: opts.composeProject, - assertFresh: opts.assertFresh, + // The launch is fenced at the mutation itself: under the instance lock, + // and only while the session still carries this start's token, so a + // displaced start never adds windows or panes to its replacement. + const started = await underInstanceLock(async () => { + if (opts.sessionOwnership && !(await sessionStillOwned())) { + relinquishDisplacedSession(); + } + return await startLifecycleProcess({ + backend: backendName as MuxBackendName, + sessionName, + projectRoot: opts.project.projectRoot, + env: opts.env, + index: nextIndex, + process: launch ? { ...process, command: launch.command } : process, + projectDir: opts.project.projectDir, + composeProject: opts.composeProject, + assertFresh: opts.assertFresh, + timeoutMs: muxTimeoutMs, + }); }); nextIndex += 1; startedProcesses.push(started); @@ -439,62 +741,162 @@ export function createLifecycleProcessController(opts: { } }, finalize: async () => { - if (startedProcesses.length === 0 || !backendName) { - await removeLifecycleStateEntry({ - projectDir: opts.project.projectDir, - composeProject: opts.composeProject, - }); + const ownership = opts.sessionOwnership; + if (!ownership) { + await publishFinalEntry(); return; } - const ownershipToken = - createdOwnershipToken ?? adoptedEntry?.ownershipToken; - await upsertLifecycleStateEntry({ - projectDir: opts.project.projectDir, - entry: { - composeProject: opts.composeProject, - projectName: opts.projectName, - branch: opts.branch, - sessionName, - backend: backendName, - ...(ownershipToken ? { ownershipToken } : {}), - definitionHash: opts.definitionHash, - processes: startedProcesses, - updatedAt: new Date().toISOString(), - }, - }); + // Conditional publication under the instance lock: another start may + // have replaced this start's pending session since it was created. + await ownership.withLock(publishFinalEntryIfStillOwned); }, abort: async () => { - await starterAbort(); + const outcome = await starterAbort(); operationSignalCleanup?.dispose(); + return outcome; }, hasStarted: () => startedProcesses.length > 0, + // Runs inside SharedHelperSessionOwnership.runIfOwned, which already + // holds the instance lock: never acquire it again here. getOperationCleanup: () => createdOwnershipToken ? async () => { - await starterAbort(); + await sessionCreationSettled; + return await abortOwnedSession(); } : null, getSignalCleanup: () => operationSignalCleanup, }; - async function starterAbort(): Promise { + /** + * Abort and signal cleanup. With shared helper ownership it takes the same + * decision as the operation cleanup (SharedHelperSessionOwnership + * .runIfOwned): under the instance lock, it stops the session only while + * the recorded session epoch is still this start's own. Adoption keeps the + * creator's token, so the token check alone would let a callback that was + * already waiting on the lock kill a session another up adopted + * meanwhile. The token read and name-based kill also run under that lock, + * as replacement does. Creation settles first: it holds the lock until its + * own failure cleanup is done. + */ + async function starterAbort(): Promise< + undefined | typeof LIFECYCLE_SESSION_RETAINED + > { await sessionCreationSettled; if (!(createdOwnershipToken && backendName)) { - return; + return undefined; + } + const ownership = opts.sessionOwnership; + if (!ownership) { + return await abortOwnedSession(); + } + let result: undefined | typeof LIFECYCLE_SESSION_RETAINED; + let outcome: Awaited>; + try { + outcome = await ownership.runIfOwned(async () => { + result = await abortOwnedSession(); + }); + } catch { + outcome = "undecidable"; + } + if (outcome === "adopted") { + // Another up adopted the session: it is no longer this start's to stop. + createdOwnershipToken = null; + return undefined; + } + if (outcome === "undecidable") { + logger.warn({ + message: `Could not confirm that lifecycle session ${sessionName} stopped; kept its ownership record. Run hack down to stop it.`, + }); + return LIFECYCLE_SESSION_RETAINED; + } + return result; + } + + /** Stop this start's created session; the caller holds the instance lock. */ + async function abortOwnedSession(): Promise< + undefined | typeof LIFECYCLE_SESSION_RETAINED + > { + if (!(createdOwnershipToken && backendName)) { + return undefined; } const backend = (await opts.mux()).backends.get(backendName); - if (backend?.available) { - await killLifecycleSessionWithOwnership({ - backend, - sessionName, - ownershipToken: createdOwnershipToken, + const stopped = backend?.available + ? await killLifecycleSessionWithOwnership({ + backend, + sessionName, + ownershipToken: createdOwnershipToken, + }).catch(() => false) + : false; + if (!stopped) { + // The session may still run: keep its ownership entry (and this + // operation's token) so a later abort, up or hack down can still prove + // ownership and reclaim it. Removing it would orphan the session. + logger.warn({ + message: `Could not confirm that lifecycle session ${sessionName} stopped; kept its ownership record. Run hack down to stop it.`, }); + return LIFECYCLE_SESSION_RETAINED; } await removeLifecycleStateEntryIfOwned({ projectDir: opts.project.projectDir, composeProject: opts.composeProject, ownershipToken: createdOwnershipToken, }); + createdOwnershipToken = null; + return undefined; + } +} + +/** + * A lifecycle start failed and its session could not be confirmed stopped: + * the session (and its ownership record) was kept. Callers must keep what + * the session depends on, such as shared helper holders. The message is the + * original failure's. + */ +export class LifecycleSessionRetainedError extends Error { + readonly original: unknown; + constructor(original: unknown) { + super( + original instanceof Error + ? original.message + : "Failed to start lifecycle processes" + ); + this.name = "LifecycleSessionRetainedError"; + this.original = original; + } +} + +/** + * Lifecycle sessions of projects with shared helpers need a backend whose + * session carries its ownership token from creation (tmux); see + * MuxBackend.createsWithOwnerToken. + */ +export const SHARED_HELPER_MUX_UNSUPPORTED_MESSAGE = + "Lifecycle sessions for projects with lifecycle.sharedHelpers need tmux; zellij cannot record the session's ownership with its creation. Install tmux or set sessions.mux to tmux. Nothing was started."; + +/** Raised by finalize when another start replaced this start's session. */ +export class LifecycleSessionDisplacedError extends Error { + constructor(sessionName: string) { + super( + `Another hack up replaced lifecycle session ${sessionName} while this one was starting; this up stopped without taking it over. Run the command again.` + ); + this.name = "LifecycleSessionDisplacedError"; + } +} + +async function applyTmuxSessionEnvironment(opts: { + readonly sessionName: string; + readonly env: Readonly>; + readonly timeoutMs: number; +}): Promise { + for (const [key, value] of Object.entries(opts.env)) { + await runMuxCommand( + ["tmux", "set-environment", "-t", opts.sessionName, key, value], + { + stdin: "ignore", + timeoutMs: opts.timeoutMs, + } + ); } } @@ -545,6 +947,7 @@ async function startLifecycleProcess(opts: { readonly projectDir: string; readonly composeProject: string; readonly assertFresh?: () => Promise; + readonly timeoutMs: number; }): Promise<{ readonly name: string; readonly windowName: string; @@ -574,7 +977,7 @@ async function startLifecycleProcess(opts: { serviceName: opts.process.name, }); await opts.assertFresh?.(); - const result = await exec( + const result = await runMuxCommand( [ "tmux", "new-window", @@ -588,7 +991,7 @@ async function startLifecycleProcess(opts: { "-c", wrappedCommand, ], - { stdin: "ignore" } + { stdin: "ignore", timeoutMs: opts.timeoutMs } ); if (result.exitCode !== 0) { await rm(commandPidPath, { force: true }); @@ -638,7 +1041,7 @@ async function startLifecycleProcess(opts: { serviceName: opts.process.name, }); await opts.assertFresh?.(); - const result = await exec( + const result = await runMuxCommand( [ "zellij", "--session", @@ -656,6 +1059,7 @@ async function startLifecycleProcess(opts: { stdin: "ignore", cwd, env: { ...opts.env }, + timeoutMs: opts.timeoutMs, } ); if (result.exitCode !== 0) { @@ -688,6 +1092,8 @@ export async function stopLifecycleProcessController(opts: { readonly composeProject: string; /** Internal frontend fence. Legacy callers retain their original owner path. */ readonly assertFresh?: () => Promise; + /** Mux backends; defaults to the installed tmux/zellij backends. */ + readonly mux?: () => Promise; }): Promise { await opts.assertFresh?.(); const sessionName = resolveLifecycleSessionName({ @@ -703,7 +1109,9 @@ export async function stopLifecycleProcessController(opts: { ) ?? null; const backend = lifecycleEntry - ? getMuxBackends().get(lifecycleEntry.backend) + ? (opts.mux ? (await opts.mux()).backends : getMuxBackends()).get( + lifecycleEntry.backend + ) : null; if (!lifecycleEntry) { return; @@ -786,7 +1194,7 @@ async function interruptLifecycleTmuxProcesses(opts: { for (const processInfo of processWindows) { await opts.assertFresh?.(); - await exec( + await runMuxCommand( [ "tmux", "send-keys", @@ -814,7 +1222,7 @@ async function readTmuxPanePids(opts: { readonly sessionName: string; readonly windowName: string; }): Promise { - const result = await exec( + const result = await runMuxCommand( [ "tmux", "list-panes", diff --git a/src/lib/project-lifecycle-sessions.ts b/src/lib/project-lifecycle-sessions.ts index 88df5e3ca..020e7bdaa 100644 --- a/src/lib/project-lifecycle-sessions.ts +++ b/src/lib/project-lifecycle-sessions.ts @@ -298,7 +298,10 @@ function hasLegacyTmuxOwnershipProof(opts: { /** A listing miss is not proof of absence: a session the listing could not * parse would otherwise be "created" again (tmux refuses the duplicate) or * skipped by down. Ask for the exact session; when it is present, return it - * without listing metadata so ownership is decided by the token alone. + * without listing metadata so ownership is decided by the token alone. An + * unknown answer (the query timed out or failed without tmux's explicit + * absence diagnostic) is never absence: the inspection fails, so down keeps + * the entry and helpers and creation refuses. */ async function confirmUnlistedSession(opts: { readonly backend: MuxBackend; @@ -307,6 +310,9 @@ async function confirmUnlistedSession(opts: { const presence = await opts.backend.readSessionPresence?.({ name: opts.name, }); + if (presence === "unknown") { + throw new LifecycleSessionPresenceUnknownError(opts.name); + } if (presence !== "present") { return null; } @@ -340,3 +346,16 @@ async function normalizePath(path: string): Promise { return path; } } + +/** + * The listing omitted the lifecycle session and its exact-presence query had + * no definite answer, so the session may still exist. + */ +export class LifecycleSessionPresenceUnknownError extends Error { + constructor(sessionName: string) { + super( + `Could not confirm whether lifecycle session "${sessionName}" exists (the mux query timed out or failed); refusing to treat it as absent. Run the command again once tmux responds.` + ); + this.name = "LifecycleSessionPresenceUnknownError"; + } +} diff --git a/src/lib/project-meta.ts b/src/lib/project-meta.ts index 466d13667..e6819c06f 100644 --- a/src/lib/project-meta.ts +++ b/src/lib/project-meta.ts @@ -1,6 +1,6 @@ import { dirname, resolve } from "node:path"; import { YAML } from "bun"; -import type { MuxSession } from "../mux/mux-backend.ts"; +import { MuxCommandTimeoutError, type MuxSession } from "../mux/mux-backend.ts"; import { getMuxBackends } from "../mux/mux-resolver.ts"; import { parseSessionBase } from "../mux/session-names.ts"; import { mergeBranchUsage, readBranchUsage } from "./branch-usage.ts"; @@ -424,7 +424,14 @@ export async function resolveSessionsMeta(opts: { if (!backend.available) { continue; } - const listed = await backend.listSessions(); + // Display only: a timed-out listing (unknown) is skipped rather than + // failing the command. Ownership decisions never use this path. + const listed = await backend.listSessions().catch((error: unknown) => { + if (error instanceof MuxCommandTimeoutError) { + return []; + } + throw error; + }); for (const s of listed) { const base = parseSessionBase({ name: s.name }); if (base === opts.projectName) { diff --git a/src/lib/project-shared-helpers.ts b/src/lib/project-shared-helpers.ts new file mode 100644 index 000000000..60e11e7e3 --- /dev/null +++ b/src/lib/project-shared-helpers.ts @@ -0,0 +1,1158 @@ +import { randomUUID } from "node:crypto"; +import { mkdir, readFile, rename, rm, stat, writeFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; + +import { resolveGlobalHackDir } from "./config-paths.ts"; +import { isRecord } from "./guards.ts"; +import { withProjectsRegistryLock } from "./projects-registry-lock.ts"; +import { + ensureSharedHostHelper, + releaseSharedHostHelper, + type SharedHostEnsureResult, + type SharedHostHelperSpec, + type SharedHostReleaseResult, + sharedHostFailureMessage, +} from "./shared-host-lifecycle.ts"; + +/** + * Project wiring for the shared host helper contract + * (docs/reference/shared-host-lifecycle.md): `lifecycle.sharedHelpers` in + * `hack.config.json`, started by `hack up`/`restart` and released by + * `hack down`. + * + * Invariants: + * - The claim directory is derived from the helper name and port only, under + * the global Hack home, so every branch and checkout of every project that + * names the same helper and port coordinates through one claim and lock. + * - The holder is the instance's Compose project (`` or + * `--`), so two branches hold one helper independently and + * the last release stops an owned helper. + * - A failed or signalled `up` releases only the holders that operation + * recorded (`holderAdded`), so a retry never drops a holder that an earlier + * successful `up` still relies on. + * - Nothing here signals a process: stopping is the library's birth-verified + * release. Messages name the helper and a closed reason, never upstreams, + * argv or environment values. + */ + +const NAME = /^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}$/; +const ENV_NAME = /^[A-Za-z_][A-Za-z0-9_]*$/; +const DEFAULT_IDENTITY_PATH = "/__hack/identity"; +const DEFAULT_READY_TIMEOUT_MS = 30_000; +const MAX_READY_TIMEOUT_MS = 600_000; +const DEFAULT_PROBE_TIMEOUT_MS = 2000; +const BACKOFF = { initialMs: 100, maxMs: 2000 } as const; +const ALLOWED_KEYS = new Set([ + "name", + "enabled", + "command", + "cwd", + "port", + "identityPath", + "upstream", + "upstreamEnv", + "env", + "readyTimeoutMs", +]); + +export type ProjectSharedHelperConfig = { + /** Helper kind: the identity `helper` value and the claim name. */ + readonly name: string; + readonly enabled: boolean; + /** Argv, executed directly (never through a shell string). */ + readonly command: readonly string[]; + readonly cwd?: string; + readonly port: number; + readonly identityPath: string; + /** Exactly one of `upstream` and `upstreamEnv` is set. */ + readonly upstream?: string; + readonly upstreamEnv?: string; + /** Names passed to the helper, besides PATH. */ + readonly env: readonly string[]; + readonly readyTimeoutMs: number; +}; + +/** + * Strictly parse `lifecycle.sharedHelpers`. Unlike other lifecycle entries, + * an invalid entry is a config error, not a silently skipped one: a helper + * that is quietly not started fails later and further from its cause. + */ +export function parseProjectSharedHelpers(value: unknown): { + readonly helpers?: readonly ProjectSharedHelperConfig[]; + readonly errors: readonly string[]; +} { + if (value === undefined) { + return { errors: [] }; + } + if (!Array.isArray(value)) { + return { errors: ["lifecycle.sharedHelpers must be an array."] }; + } + const errors: string[] = []; + const helpers: ProjectSharedHelperConfig[] = []; + value.forEach((entry: unknown, index: number) => { + const parsed = parseHelper(entry, `lifecycle.sharedHelpers[${index}]`); + if ("error" in parsed) { + errors.push(parsed.error); + return; + } + if (helpers.some((helper) => helper.name === parsed.helper.name)) { + errors.push( + `lifecycle.sharedHelpers[${index}].name duplicates "${parsed.helper.name}".` + ); + return; + } + if (helpers.some((helper) => helper.port === parsed.helper.port)) { + errors.push( + `lifecycle.sharedHelpers[${index}].port duplicates ${parsed.helper.port}.` + ); + return; + } + helpers.push(parsed.helper); + }); + return { + ...(helpers.length > 0 && errors.length === 0 ? { helpers } : {}), + errors, + }; +} + +// biome-ignore lint/complexity/noExcessiveCognitiveComplexity: One strict field-by-field validator keeps every refusal message next to its rule. +function parseHelper( + value: unknown, + at: string +): { readonly helper: ProjectSharedHelperConfig } | { readonly error: string } { + if (!isRecord(value)) { + return { error: `${at} must be an object.` }; + } + const unknownKey = Object.keys(value).find((key) => !ALLOWED_KEYS.has(key)); + if (unknownKey !== undefined) { + return { error: `${at} has unknown key "${unknownKey}".` }; + } + const name = value.name; + if (typeof name !== "string" || !NAME.test(name)) { + return { + error: `${at}.name must match [A-Za-z0-9][A-Za-z0-9_.-]{0,127}.`, + }; + } + const command = value.command; + if ( + !Array.isArray(command) || + command.length === 0 || + !command.every((part) => typeof part === "string" && part.length > 0) + ) { + return { + error: `${at}.command must be a non-empty array of non-empty strings (argv, not a shell string).`, + }; + } + const port = value.port; + if ( + typeof port !== "number" || + !Number.isInteger(port) || + port < 1 || + port > 65_535 + ) { + return { error: `${at}.port must be an integer from 1 to 65535.` }; + } + if (value.enabled !== undefined && typeof value.enabled !== "boolean") { + return { error: `${at}.enabled must be a boolean.` }; + } + if ( + value.cwd !== undefined && + (typeof value.cwd !== "string" || value.cwd.trim().length === 0) + ) { + return { error: `${at}.cwd must be a non-empty string.` }; + } + const identityPath = value.identityPath ?? DEFAULT_IDENTITY_PATH; + if (typeof identityPath !== "string" || !identityPath.startsWith("/")) { + return { error: `${at}.identityPath must be an absolute path.` }; + } + const hasUpstream = value.upstream !== undefined; + const hasUpstreamEnv = value.upstreamEnv !== undefined; + if (hasUpstream === hasUpstreamEnv) { + return { error: `${at} needs exactly one of upstream and upstreamEnv.` }; + } + if ( + hasUpstream && + (typeof value.upstream !== "string" || value.upstream.length === 0) + ) { + return { error: `${at}.upstream must be a non-empty string.` }; + } + if ( + hasUpstreamEnv && + (typeof value.upstreamEnv !== "string" || !ENV_NAME.test(value.upstreamEnv)) + ) { + return { error: `${at}.upstreamEnv must be an environment variable name.` }; + } + const env = value.env ?? []; + if ( + !( + Array.isArray(env) && + env.every((entry) => typeof entry === "string" && ENV_NAME.test(entry)) + ) || + new Set(env).size !== env.length + ) { + return { + error: `${at}.env must be an array of unique environment variable names.`, + }; + } + const readyTimeoutMs = value.readyTimeoutMs ?? DEFAULT_READY_TIMEOUT_MS; + if ( + typeof readyTimeoutMs !== "number" || + !Number.isInteger(readyTimeoutMs) || + readyTimeoutMs < 1 || + readyTimeoutMs > MAX_READY_TIMEOUT_MS + ) { + return { + error: `${at}.readyTimeoutMs must be an integer from 1 to ${MAX_READY_TIMEOUT_MS}.`, + }; + } + return { + helper: { + name, + enabled: value.enabled !== false, + command: command as string[], + ...(typeof value.cwd === "string" ? { cwd: value.cwd.trim() } : {}), + port, + identityPath, + ...(hasUpstream ? { upstream: value.upstream as string } : {}), + ...(hasUpstreamEnv ? { upstreamEnv: value.upstreamEnv as string } : {}), + env: env as string[], + readyTimeoutMs, + }, + }; +} + +/** + * The claim directory for one helper. It depends on the helper name and port + * only, never on the project, branch or checkout, so every client of that + * helper port shares one claim and lock. + */ +export function resolveSharedHelperStateDir(opts: { + readonly hackDir?: string; + readonly name: string; + readonly port: number; +}): string { + return join( + opts.hackDir ?? resolveGlobalHackDir(), + "shared-helpers", + `${opts.name}-${opts.port}` + ); +} + +/** Holder for one project instance; refuses names the claim cannot record. */ +export function resolveSharedHelperHolder(composeProject: string): string { + if (!NAME.test(composeProject)) { + throw new Error( + "Shared helpers need a Compose project name of letters, digits, '.', '_' or '-'." + ); + } + return composeProject; +} + +/** + * Build the library spec. The helper sees PATH from the CLI plus only the + * names it lists; each comes from the resolved lifecycle env, then the CLI + * environment. A listed name that is set in neither is an error. + */ +export function buildSharedHelperSpec(opts: { + readonly helper: ProjectSharedHelperConfig; + readonly projectRoot: string; + readonly env: Readonly>; + readonly processEnv?: Readonly>; + readonly hackDir?: string; +}): SharedHostHelperSpec { + const { helper } = opts; + const processEnv = opts.processEnv ?? process.env; + const lookup = (name: string): string | undefined => + opts.env[name] ?? processEnv[name]; + const childEnv: Record = {}; + const path = processEnv.PATH; + if (path !== undefined) { + childEnv.PATH = path; + } + for (const name of helper.env) { + const value = lookup(name); + if (value === undefined) { + throw new Error( + `Shared helper "${helper.name}" lists env ${name}, but it is not set.` + ); + } + childEnv[name] = value; + } + const upstream = + helper.upstream ?? + (helper.upstreamEnv === undefined ? undefined : lookup(helper.upstreamEnv)); + if (upstream === undefined || upstream.length === 0) { + throw new Error( + `Shared helper "${helper.name}" reads its upstream from ${helper.upstreamEnv}, but it is not set.` + ); + } + const cwd = helper.cwd + ? resolve(opts.projectRoot, helper.cwd) + : opts.projectRoot; + return { + name: helper.name, + enabled: helper.enabled, + stateDir: resolveSharedHelperStateDir({ + ...(opts.hackDir ? { hackDir: opts.hackDir } : {}), + name: helper.name, + port: helper.port, + }), + port: helper.port, + identityPath: helper.identityPath, + expected: { helper: helper.name, upstream }, + command: helper.command, + cwd, + env: childEnv, + deadlineMs: helper.readyTimeoutMs, + backoff: BACKOFF, + probeTimeoutMs: Math.min(DEFAULT_PROBE_TIMEOUT_MS, helper.readyTimeoutMs), + }; +} + +export type SharedHelperStartRecord = { + readonly name: string; + readonly result: SharedHostEnsureResult; +}; + +export type SharedHelperRelease = { + readonly name: string; + readonly result: SharedHostReleaseResult; +}; + +/** Raised when a shared helper is not ready; holders added by this run were released first. */ +export class SharedHelperStartError extends Error { + readonly helper: string; + readonly reason: string; + constructor(opts: { + readonly helper: string; + readonly reason: string; + readonly message: string; + }) { + super( + `Shared helper "${opts.helper}" failed (${opts.reason}): ${opts.message}` + ); + this.name = "SharedHelperStartError"; + this.helper = opts.helper; + this.reason = opts.reason; + } +} + +/** Raised when a concurrent `hack down` superseded this start. */ +export class SharedHelperSupersededError extends Error { + /** Whether services were already launched when the down was observed. */ + readonly servicesStarted: boolean; + constructor(opts: { readonly servicesStarted?: boolean } = {}) { + super( + opts.servicesStarted === true + ? "A concurrent hack down released this instance's shared helpers while its services started; run hack down to stop them, then hack up again." + : "A concurrent hack down released this instance's shared helpers; startup stopped before launching services that need them. Run hack up again." + ); + this.name = "SharedHelperSupersededError"; + this.servicesStarted = opts.servicesStarted === true; + } +} + +/** + * Ensure every configured helper for one instance, in config order. + * + * On the first failure (or a thrown spec error) it releases, in reverse + * order, the holders this call added, then throws. The returned `rollback` + * does the same after a later phase of the operation fails. + * + * Instance coordination (all under one per-instance lock): + * - A start whose helpers are all ready records its operation id as the + * instance's current operation (a start that fails never does). + * A rollback releases only while that is still current, so it never revokes + * a newer start of the same instance. + * - Shutdown is tracked separately, as the instance's shutdown generation. + * Only `hack down` advances it: its stopping mark + * ({@link markSharedHelperInstanceStopping}) and its release. A start + * captures the generation, and `assertCurrent` fails once it changed, so + * the caller can refuse to launch services whose helpers a down released. + * A later start never restores it, even a successful one: its helpers need + * not satisfy this start's (an entry may be disabled or changed since). A + * newer start without a down in between does not fail it. + * - The generation is the one the caller captured before its preparation + * hooks ran ({@link captureSharedHelperShutdownGeneration}), so a down + * during those hooks also supersedes this start: it fails before ensuring + * any helper. + * - Lifecycle-session cleanup is coordinated separately, by + * {@link createSharedHelperSessionOwnership}. + * - Before each helper is ensured its identity (name and port) is added to + * the instance's acquired record, so `hack down` still releases it after the + * entry is removed or renamed in config. + */ +export async function startProjectSharedHelpers(opts: { + readonly helpers: readonly ProjectSharedHelperConfig[] | undefined; + readonly projectRoot: string; + readonly composeProject: string; + readonly env: Readonly>; + readonly signal?: AbortSignal; + readonly hackDir?: string; + readonly processEnv?: Readonly>; + readonly report?: (record: SharedHelperStartRecord) => void | Promise; + /** + * The shutdown generation captured when the operation began, before its + * preparation hooks. Omitted, it is read when helper startup begins. + */ + readonly shutdownGeneration?: SharedHelperShutdownGeneration; +}): Promise<{ + readonly started: readonly SharedHelperStartRecord[]; + readonly rollback: () => Promise; + /** Throws {@link SharedHelperSupersededError} once a down superseded this start. */ + readonly assertCurrent: (opts?: { + readonly servicesStarted?: boolean; + }) => Promise; +}> { + const helpers = opts.helpers ?? []; + const started: SharedHelperStartRecord[] = []; + const added: SharedHostHelperSpec[] = []; + if (helpers.length === 0) { + return { + started, + rollback: () => Promise.resolve([]), + assertCurrent: () => Promise.resolve(), + }; + } + const holder = resolveSharedHelperHolder(opts.composeProject); + const instance = instancePaths({ hackDir: opts.hackDir, holder }); + const operation = randomUUID(); + // Captured under the lock before the first ensure; see assertCurrent. + let generation: string | null = null; + // Each ensure is bounded by its own deadline; the lock wait covers them all. + const lockTimeoutMs = + helpers.reduce((sum, helper) => sum + helper.readyTimeoutMs, 0) + 10_000; + const releaseAdded = (): Promise => + releaseSpecs({ specs: [...added].reverse(), holder }); + let rolledBack: Promise | null = null; + const rollback = (): Promise => { + rolledBack ??= + added.length === 0 + ? Promise.resolve([]) + : withInstanceLock({ + instance, + timeoutMs: INSTANCE_LOCK_TIMEOUT_MS, + run: async () => { + // Release only while this start is still the instance's + // current operation and no down has advanced the shutdown + // generation since it began. A down publishes the generation + // and then its operation record as two writes, so a down + // interrupted between them leaves this start's record in + // place: the generation check keeps the holders anyway (the + // down, or the next one, releases them), never stopping a + // helper underneath containers this start already launched. + if ( + (await readOperation(instance)) !== operation || + (await readGeneration(instance)) !== generation + ) { + return []; + } + await rm(instance.operationPath, { force: true }); + return await releaseAdded(); + }, + }).catch( + // The lock was unavailable: whether this start is still current + // cannot be decided, so keep its holders (hack down releases + // them) and report it rather than throw from cleanup. + (): readonly SharedHelperRelease[] => + added.map((spec) => ({ + name: spec.name, + result: { + state: "failed", + reason: "lock-unavailable", + message: sharedHostFailureMessage("lock-unavailable"), + }, + })) + ); + return rolledBack; + }; + const assertCurrent = async ( + check: { readonly servicesStarted?: boolean } = {} + ): Promise => { + const current = await withInstanceLock({ + instance, + timeoutMs: INSTANCE_LOCK_TIMEOUT_MS, + run: async () => (await readGeneration(instance)) === generation, + }); + if (!current) { + throw new SharedHelperSupersededError(check); + } + }; + await mkdir(instance.dir, { recursive: true, mode: 0o700 }); + await withInstanceLock({ + instance, + timeoutMs: lockTimeoutMs, + ...(opts.signal ? { signal: opts.signal } : {}), + run: async () => { + const current = await readGeneration(instance); + if ( + opts.shutdownGeneration !== undefined && + current !== opts.shutdownGeneration.value + ) { + // A down ran after this operation began (during its preparation + // hooks): never start helpers for it. + throw new SharedHelperSupersededError(); + } + generation = current; + try { + for (const helper of helpers) { + if (helper.enabled) { + await recordAcquired(instance, [ + { name: helper.name, port: helper.port }, + ]); + } + const result = await ensureOne({ ...opts, helper, holder }); + if (result.state === "ready" && result.holderAdded) { + added.push(result.spec); + } + const record = { name: helper.name, result: result.result }; + started.push(record); + await opts.report?.(record); + if (record.result.state === "failed") { + throw new SharedHelperStartError({ + helper: helper.name, + reason: record.result.reason, + message: record.result.message, + }); + } + } + // Only a start whose helpers are all ready becomes the current + // operation. Recording earlier would let a start that then fails + // replace a down's shutdown mark, so an older start would launch its + // services without helpers. A failed write rolls back like any + // other failure. + await writeOperation(instance, operation); + } catch (error: unknown) { + // Still under the instance lock, and this start never became the + // current operation: release exactly what it added. The operation + // record is left as it was, so a shutdown mark stays in force. + rolledBack = releaseAdded(); + await rolledBack; + throw error; + } + }, + }); + return { started, rollback, assertCurrent }; +} + +async function ensureOne(opts: { + readonly helper: ProjectSharedHelperConfig; + readonly holder: string; + readonly projectRoot: string; + readonly env: Readonly>; + readonly signal?: AbortSignal; + readonly hackDir?: string; + readonly processEnv?: Readonly>; +}): Promise< + | { + readonly state: "ready"; + readonly holderAdded: boolean; + readonly spec: SharedHostHelperSpec; + readonly result: SharedHostEnsureResult; + } + | { readonly state: "other"; readonly result: SharedHostEnsureResult } +> { + // A disabled helper needs none of its runtime env: it is never started. + if (!opts.helper.enabled) { + return { + state: "other", + result: { state: "disabled", ownership: "disabled" }, + }; + } + const spec = buildSharedHelperSpec({ + helper: opts.helper, + projectRoot: opts.projectRoot, + env: opts.env, + ...(opts.processEnv ? { processEnv: opts.processEnv } : {}), + ...(opts.hackDir ? { hackDir: opts.hackDir } : {}), + }); + const result = await ensureSharedHostHelper({ + spec, + holder: opts.holder, + ...(opts.signal ? { signal: opts.signal } : {}), + }); + return result.state === "ready" + ? { state: "ready", holderAdded: result.holderAdded, spec, result } + : { state: "other", result }; +} + +const INSTANCE_LOCK_TIMEOUT_MS = 10_000; +const STOPPING_PREFIX = "stopping:"; +/** + * Failure name used when this instance's helpers cannot be listed (its + * acquired record is unreadable or its lock is unavailable) and no helper is + * configured to name instead: the failure must still reach `hack down`. + */ +const UNLISTED_HELPERS = "recorded-helpers"; +const MAX_ACQUIRED = 256; + +type HelperIdentity = { readonly name: string; readonly port: number }; + +type InstancePaths = { + readonly dir: string; + readonly lockPath: string; + readonly operationPath: string; + readonly generationPath: string; + readonly sessionEpochPath: string; + readonly acquiredPath: string; +}; + +function instancePaths(opts: { + readonly hackDir?: string; + readonly holder: string; +}): InstancePaths { + const dir = join( + opts.hackDir ?? resolveGlobalHackDir(), + "shared-helpers", + ".instances" + ); + return { + dir, + lockPath: join(dir, `${opts.holder}.lock`), + operationPath: join(dir, `${opts.holder}.operation`), + generationPath: join(dir, `${opts.holder}.shutdown-generation`), + sessionEpochPath: join(dir, `${opts.holder}.session-epoch`), + acquiredPath: join(dir, `${opts.holder}.acquired.json`), + }; +} + +async function withInstanceLock(opts: { + readonly instance: InstancePaths; + readonly timeoutMs: number; + readonly signal?: AbortSignal; + readonly run: () => Promise; +}): Promise { + return await withProjectsRegistryLock({ + lockPath: opts.instance.lockPath, + label: "shared helper instance", + timeoutMs: opts.timeoutMs, + ...(opts.signal ? { signal: opts.signal } : {}), + run: opts.run, + }); +} + +async function readOperation(instance: InstancePaths): Promise { + try { + return (await readFile(instance.operationPath, "utf8")).trim(); + } catch { + return null; + } +} + +/** Outcome of {@link SharedHelperSessionOwnership.runIfOwned}. */ +export type ProcessCleanupOutcome = + /** The cleanup ran. */ + | "ran" + /** Another operation adopted (or replaced) the session since: kept. */ + | "adopted" + /** The instance lock was unavailable, so ownership is unknown: kept. */ + | "undecidable"; + +/** + * Lifecycle-session ownership for one operation of an instance with shared + * helpers. + * + * The lifecycle session keeps its creator's ownership token when another + * `hack up` adopts it, so the token alone cannot tell the creator whether the + * session is still only its own. Each session decision is recorded instead: + * under the instance lock, adopting writes a fresh session epoch, and + * creating writes one that the creator remembers. The creator's cleanup kills + * the session only while the epoch is still the one it wrote, checked under + * the same lock, so no adoption can slip between the check and the kill and + * no creation can overwrite an adoption's record. This is independent of the + * order in which operations ensured their helpers. + */ +export type SharedHelperSessionOwnership = { + /** Run a session decision or creation under the instance lock. */ + readonly withLock: (run: () => Promise) => Promise; + /** Record a decision; call inside {@link withLock}. */ + readonly record: (kind: "adopted" | "created") => Promise; + /** Run this operation's session cleanup only while it still owns the session. */ + readonly runIfOwned: ( + cleanup: () => Promise + ) => Promise; +}; + +export function createSharedHelperSessionOwnership(opts: { + readonly composeProject: string; + readonly hackDir?: string; + /** Lock wait for a session decision; another start may hold it while ensuring helpers. */ + readonly decisionTimeoutMs: number; +}): SharedHelperSessionOwnership { + const holder = resolveSharedHelperHolder(opts.composeProject); + const instance = instancePaths({ hackDir: opts.hackDir, holder }); + let createdEpoch: string | null = null; + const readEpoch = async (): Promise => { + try { + return (await readFile(instance.sessionEpochPath, "utf8")).trim(); + } catch { + return null; + } + }; + return { + withLock: async (run) => { + await mkdir(instance.dir, { recursive: true, mode: 0o700 }); + return await withInstanceLock({ + instance, + timeoutMs: opts.decisionTimeoutMs, + run, + }); + }, + record: async (kind) => { + const epoch = randomUUID(); + await writeFileAtomic(instance.sessionEpochPath, `${epoch}\n`); + if (kind === "created") { + createdEpoch = epoch; + } + }, + runIfOwned: async (cleanup) => { + // Always under the lock, even when this operation never recorded a + // created session (creation failed before its record): the cleanup + // reads the session token and kills by name, and a replacement under + // the lock must not run between the two. + let decided = false; + try { + return await withInstanceLock({ + instance, + timeoutMs: INSTANCE_LOCK_TIMEOUT_MS, + run: async (): Promise => { + decided = true; + if (createdEpoch !== null && (await readEpoch()) !== createdEpoch) { + return "adopted"; + } + await cleanup(); + return "ran"; + }, + }); + } catch (error: unknown) { + if (decided) { + throw error; + } + // The lock was unavailable: ownership cannot be decided. Keep the + // session (hack down stops it) rather than risk killing an adopted + // one with the still-valid creator token. + return "undecidable"; + } + }, + }; +} + +/** An instance's shutdown generation, captured when an operation begins. */ +export type SharedHelperShutdownGeneration = { + readonly value: string | null; +}; + +/** + * Capture the instance's shutdown generation when an operation begins, before + * any preparation hook runs. Pass it to {@link startProjectSharedHelpers}: a + * down that completes while those hooks run then supersedes the operation. + * The record is replaced atomically, so it is read without the lock. + */ +export async function captureSharedHelperShutdownGeneration(opts: { + readonly composeProject: string; + readonly hackDir?: string; +}): Promise { + const holder = resolveSharedHelperHolder(opts.composeProject); + return { + value: await readGeneration( + instancePaths({ hackDir: opts.hackDir, holder }) + ), + }; +} + +/** The instance's shutdown generation; null until the first `hack down`. */ +async function readGeneration(instance: InstancePaths): Promise { + try { + return (await readFile(instance.generationPath, "utf8")).trim(); + } catch { + return null; + } +} + +/** Advance the shutdown generation; only `hack down` calls this. */ +async function advanceGeneration(instance: InstancePaths): Promise { + await writeFileAtomic(instance.generationPath, `${randomUUID()}\n`); +} + +async function writeOperation( + instance: InstancePaths, + operation: string +): Promise { + await writeFileAtomic(instance.operationPath, `${operation}\n`); +} + +async function writeFileAtomic(path: string, text: string): Promise { + const staging = `${path}.${randomUUID()}.tmp`; + await writeFile(staging, text, { mode: 0o600 }); + await rename(staging, path); +} + +/** + * Helper identities this instance may hold. A missing file is empty; an + * unreadable or malformed one throws, so `hack down` fails closed instead of + * silently forgetting a holder. + */ +async function readAcquired( + instance: InstancePaths +): Promise { + let text: string; + try { + text = await readFile(instance.acquiredPath, "utf8"); + } catch (error: unknown) { + if (isRecord(error) && error.code === "ENOENT") { + return []; + } + throw error; + } + const value: unknown = JSON.parse(text); + if ( + !Array.isArray(value) || + value.length > MAX_ACQUIRED || + !value.every( + (entry: unknown) => + isRecord(entry) && + Object.keys(entry).length === 2 && + typeof entry.name === "string" && + NAME.test(entry.name) && + typeof entry.port === "number" && + Number.isInteger(entry.port) && + entry.port >= 1 && + entry.port <= 65_535 + ) + ) { + throw new Error("Invalid shared helper instance record"); + } + return value as HelperIdentity[]; +} + +async function writeAcquired( + instance: InstancePaths, + identities: readonly HelperIdentity[] +): Promise { + if (identities.length === 0) { + await rm(instance.acquiredPath, { force: true }); + return; + } + await writeFileAtomic( + instance.acquiredPath, + `${JSON.stringify(identities)}\n` + ); +} + +function sameIdentity(left: HelperIdentity, right: HelperIdentity): boolean { + return left.name === right.name && left.port === right.port; +} + +function mergeIdentities( + ...lists: readonly (readonly HelperIdentity[])[] +): HelperIdentity[] { + const out: HelperIdentity[] = []; + for (const entry of lists.flat()) { + if (!out.some((existing) => sameIdentity(existing, entry))) { + out.push({ name: entry.name, port: entry.port }); + } + } + return out; +} + +async function recordAcquired( + instance: InstancePaths, + identities: readonly HelperIdentity[] +): Promise { + const current = await readAcquired(instance); + const merged = mergeIdentities(current, identities); + if (merged.length > MAX_ACQUIRED) { + // Never publish a record readAcquired would reject: that would make + // every later down fail without releasing anything. + throw new Error( + `This instance already records ${MAX_ACQUIRED} shared helpers; run hack down to release them before adding another.` + ); + } + if (merged.length !== current.length) { + await writeAcquired(instance, merged); + } +} + +function holderIfValid(composeProject: string): string | null { + return NAME.test(composeProject) ? composeProject : null; +} + +async function instanceHasState(instance: InstancePaths): Promise { + try { + await stat(instance.acquiredPath); + return true; + } catch { + return (await readOperation(instance)) !== null; + } +} + +/** + * Mark the instance as stopping before `hack down` stops anything, so a start + * still in flight fails its {@link startProjectSharedHelpers} `assertCurrent` + * check instead of launching services whose helpers this down releases. + * Returns false when the mark could not be written (the down continues; its + * release still runs under the instance lock). + */ +export async function markSharedHelperInstanceStopping(opts: { + readonly helpers: readonly ProjectSharedHelperConfig[] | undefined; + readonly composeProject: string; + readonly hackDir?: string; +}): Promise { + const holder = holderIfValid(opts.composeProject); + if (holder === null) { + return true; + } + const instance = instancePaths({ hackDir: opts.hackDir, holder }); + if ( + (opts.helpers ?? []).length === 0 && + !(await instanceHasState(instance)) + ) { + return true; + } + try { + await mkdir(instance.dir, { recursive: true, mode: 0o700 }); + await withInstanceLock({ + instance, + timeoutMs: INSTANCE_LOCK_TIMEOUT_MS, + run: async () => { + await advanceGeneration(instance); + await writeOperation(instance, `${STOPPING_PREFIX}${randomUUID()}`); + }, + }); + return true; + } catch { + return false; + } +} + +/** + * Release this instance's holder from every helper it may hold: the ones + * recorded as acquired (so a removed or renamed config entry is still + * released) plus every configured one, enabled or not. Every helper is + * attempted; failures are returned, not thrown, and a failed helper stays in + * the acquired record so the next down retries it. + */ +/** + * Names of the helpers this instance may still hold: the configured entries + * plus those in its acquired record (entries removed from config since they + * were acquired). Read without the lock, for reporting what a down keeps; an + * unreadable record yields a placeholder name so the caller still fails. + * Never throws. + */ +export async function listSharedHelpersHeld(opts: { + readonly helpers: readonly ProjectSharedHelperConfig[] | undefined; + readonly composeProject: string; + readonly hackDir?: string; +}): Promise { + const configured = (opts.helpers ?? []).map((helper) => ({ + name: helper.name, + port: helper.port, + })); + const holder = holderIfValid(opts.composeProject); + if (holder === null) { + return configured.map((identity) => identity.name); + } + const instance = instancePaths({ hackDir: opts.hackDir, holder }); + let recorded: readonly HelperIdentity[]; + try { + recorded = await readAcquired(instance); + } catch { + recorded = [{ name: UNLISTED_HELPERS, port: 0 }]; + } + return [ + ...new Set( + mergeIdentities(configured, recorded).map((identity) => identity.name) + ), + ]; +} + +export async function releaseProjectSharedHelpers(opts: { + readonly helpers: readonly ProjectSharedHelperConfig[] | undefined; + readonly composeProject: string; + readonly hackDir?: string; +}): Promise { + const helpers = opts.helpers ?? []; + const holder = holderIfValid(opts.composeProject); + if (holder === null) { + if (helpers.length > 0) { + // Throws the same explicit refusal start uses. + resolveSharedHelperHolder(opts.composeProject); + } + return []; + } + const instance = instancePaths({ hackDir: opts.hackDir, holder }); + if (helpers.length === 0 && !(await instanceHasState(instance))) { + return []; + } + const failedAll = ( + identities: readonly HelperIdentity[], + reason: "lock-unavailable" | "state-unavailable" + ): SharedHelperRelease[] => + (identities.length > 0 + ? identities + : [{ name: UNLISTED_HELPERS, port: 0 }] + ).map((identity) => ({ + name: identity.name, + result: { + state: "failed", + reason, + message: sharedHostFailureMessage(reason), + }, + })); + const configured = helpers.map((helper) => ({ + name: helper.name, + port: helper.port, + })); + // The instance lock serializes this release with a start of the same + // instance, so up and down can never split the instance's helper set (up + // holding one helper while down stops another). Clearing the operation + // record also keeps an earlier start's rollback from acting after this. + try { + await mkdir(instance.dir, { recursive: true, mode: 0o700 }); + return await withInstanceLock({ + instance, + timeoutMs: + helpers.reduce((sum, helper) => sum + helper.readyTimeoutMs, 0) + + INSTANCE_LOCK_TIMEOUT_MS, + run: async () => { + let recorded: readonly HelperIdentity[]; + try { + recorded = await readAcquired(instance); + } catch { + return failedAll(configured, "state-unavailable"); + } + await advanceGeneration(instance); + await rm(instance.operationPath, { force: true }); + const targets = mergeIdentities(configured, recorded).reverse(); + const releases = await releaseSpecs({ + specs: targets.map((identity) => ({ + name: identity.name, + stateDir: resolveSharedHelperStateDir({ + ...(opts.hackDir ? { hackDir: opts.hackDir } : {}), + name: identity.name, + port: identity.port, + }), + })), + holder, + }); + await writeAcquired( + instance, + targets.filter((_, index) => { + const release = releases[index]; + return ( + release !== undefined && isSharedHelperReleaseProblem(release) + ); + }) + ); + return releases; + }, + }); + } catch { + return failedAll(configured, "lock-unavailable"); + } +} + +async function releaseSpecs(opts: { + readonly specs: readonly Pick[]; + readonly holder: string; +}): Promise { + const out: SharedHelperRelease[] = []; + for (const spec of opts.specs) { + out.push({ + name: spec.name, + result: await releaseSharedHostHelper({ spec, holder: opts.holder }), + }); + } + return out; +} + +/** True when a release left a helper or its claim in a state that needs an operator. */ +export function isSharedHelperReleaseProblem( + release: SharedHelperRelease +): boolean { + return ( + release.result.state === "failed" || + release.result.state === "stop-unconfirmed" + ); +} + +/** One value-free line for a release outcome. */ +export function describeSharedHelperRelease( + release: SharedHelperRelease +): string { + const result = release.result; + if (result.state === "failed") { + return `Shared helper "${release.name}" was not released (${result.reason}): ${result.message}`; + } + switch (result.state) { + case "absent": + return `Shared helper "${release.name}": no claim to release.`; + case "retained": + return `Shared helper "${release.name}" left running for its other holders.`; + case "released-adopted": + return `Shared helper "${release.name}" released; it was adopted, so Hack left it running.`; + case "stale-cleared": + return `Shared helper "${release.name}": stale claim cleared; nothing was signalled.`; + case "stopped": + return `Shared helper "${release.name}" stopped.`; + default: + return `Shared helper "${release.name}" did not confirm its stop; its claim was kept for inspection.`; + } +} + +/** One value-free line for a ready or disabled start. */ +export function describeSharedHelperStart( + record: SharedHelperStartRecord +): string { + const result = record.result; + if (result.state === "disabled") { + return `Shared helper "${record.name}" is disabled; not started.`; + } + if (result.state === "failed") { + return `Shared helper "${record.name}" failed (${result.reason}): ${result.message}`; + } + let how = "reused"; + if (result.ownership === "adopted") { + how = "adopted (identity verified; Hack will not stop it)"; + } else if (result.spawned) { + how = "started"; + } + return `Shared helper "${record.name}" ${how} at ${result.endpoint}.`; +} + +/** + * Run `run` with an AbortSignal that SIGINT/SIGTERM trips. The handlers stay + * installed until `run` settles. If a signal was caught, `onSignalled` then + * runs (the caller's cleanup for whatever `run` started), the handlers are + * removed and the signal is re-raised, so the process still ends the way the + * sender asked. + */ +export async function withTerminationAbort( + run: (signal: AbortSignal) => Promise, + opts: { readonly onSignalled?: () => Promise } = {} +): Promise { + const controller = new AbortController(); + let caught: NodeJS.Signals | null = null; + const handlers = new Map void>(); + for (const name of ["SIGINT", "SIGTERM"] as const) { + const handler = (): void => { + caught ??= name; + controller.abort(); + }; + handlers.set(name, handler); + process.on(name, handler); + } + try { + return await run(controller.signal); + } finally { + if (caught !== null) { + await opts.onSignalled?.().catch(() => undefined); + } + for (const [name, handler] of handlers) { + process.off(name, handler); + } + if (caught !== null) { + process.kill(process.pid, caught); + } + } +} diff --git a/src/lib/project.ts b/src/lib/project.ts index a6b223413..ebf6ce32a 100644 --- a/src/lib/project.ts +++ b/src/lib/project.ts @@ -24,6 +24,10 @@ import { requireLegacyProjectInputs, } from "./project-input-selection.ts"; import { normalizeProjectName } from "./project-name.ts"; +import { + type ProjectSharedHelperConfig, + parseProjectSharedHelpers, +} from "./project-shared-helpers.ts"; export type ProjectDirName = | typeof HACK_PROJECT_DIR_PRIMARY @@ -242,6 +246,12 @@ export interface ProjectConfig { readonly ownership: ProjectOwnershipConfig; readonly configPath?: string; readonly parseError?: string; + /** + * Set when `lifecycle.sharedHelpers` is invalid. Startup must refuse it + * rather than run without helpers: several Compose commands only warn on + * `parseError`, which would otherwise launch services with no helper. + */ + readonly sharedHelpersError?: string; } export interface ProjectOpenConfig { @@ -280,6 +290,8 @@ export interface ProjectLifecycleConfig { readonly up?: ProjectLifecycleHooks; readonly down?: ProjectLifecycleHooks; readonly processes?: readonly ProjectLifecycleProcess[]; + /** Identity-verified shared host helpers; see project-shared-helpers.ts. */ + readonly sharedHelpers?: readonly ProjectSharedHelperConfig[]; } export type ProjectLifecycleSingletonConflictPolicy = "adopt" | "fail"; @@ -536,12 +548,19 @@ function parseProjectConfigRecord(value: unknown, path: string): ProjectConfig { const internal = parseInternalConfig(getRecord(value, "internal")); const worktree = parseWorktreeConfig(getRecord(value, "worktree")); const sessions = parseSessionsConfig(getRecord(value, "sessions")); - const lifecycleBase = parseLifecycleConfig(getRecord(value, "lifecycle")); + const lifecycleRecord = getRecord(value, "lifecycle"); + const lifecycleBase = parseLifecycleConfig(lifecycleRecord); + const sharedHelpers = parseProjectSharedHelpers( + lifecycleRecord?.sharedHelpers + ); const startup = parseStartupEntries(value.startup); - const lifecycle = mergeLifecycleWithStartup({ + const lifecycleMerged = mergeLifecycleWithStartup({ lifecycle: lifecycleBase, startup, }); + const lifecycle = sharedHelpers.helpers + ? { ...lifecycleMerged, sharedHelpers: sharedHelpers.helpers } + : lifecycleMerged; const parsedOwnership = parseProjectOwnership(value.ownership); const defaultEnvConfig = typeof defaultEnvConfigRaw === "string" @@ -564,6 +583,7 @@ function parseProjectConfigRecord(value: unknown, path: string): ProjectConfig { : []), ...(parsedOwnership.parseError ? [parsedOwnership.parseError] : []), ...(parsedOpen.parseError ? [parsedOpen.parseError] : []), + ...sharedHelpers.errors, ]; return { @@ -582,6 +602,9 @@ function parseProjectConfigRecord(value: unknown, path: string): ProjectConfig { ...(lifecycle ? { lifecycle } : {}), ownership: parsedOwnership.ownership, ...(parseErrors.length > 0 ? { parseError: parseErrors.join(" ") } : {}), + ...(sharedHelpers.errors.length > 0 + ? { sharedHelpersError: sharedHelpers.errors.join(" ") } + : {}), configPath: path, }; } diff --git a/src/lib/shared-host-lifecycle.ts b/src/lib/shared-host-lifecycle.ts index 053157130..dad499c48 100644 --- a/src/lib/shared-host-lifecycle.ts +++ b/src/lib/shared-host-lifecycle.ts @@ -157,6 +157,12 @@ export type SharedHostEnsureResult = /** Present for owned helpers only. */ readonly pid?: number; readonly spawned: boolean; + /** + * True when this call recorded `holder` (it was not in the claim before). + * Decided under the helper lock, so a caller that rolls back a failed + * operation releases only the holder that operation added. + */ + readonly holderAdded: boolean; } | { readonly state: "failed"; @@ -886,6 +892,7 @@ async function reuseOwned( endpoint: endpointFor(spec.port), pid: claim.pid, spawned: false, + holderAdded: !claim.holders.includes(holder), }; } @@ -997,6 +1004,7 @@ async function adoptOrRefuse( ownership: "adopted", endpoint: endpointFor(spec.port), spawned: false, + holderAdded: !carried.includes(holder), }; default: return null; @@ -1097,6 +1105,7 @@ async function spawnOwned( endpoint: endpointFor(spec.port), pid: identity.pid, spawned: true, + holderAdded: !carried.includes(holder), }; } catch (error: unknown) { // An observer or a state write failed. Never leave a gate waiting while diff --git a/src/mux/mux-backend.ts b/src/mux/mux-backend.ts index 28d316f29..e9f17d54a 100644 --- a/src/mux/mux-backend.ts +++ b/src/mux/mux-backend.ts @@ -1,4 +1,4 @@ -import type { ExecResult } from "../lib/shell.ts"; +import { type ExecOptions, type ExecResult, exec } from "../lib/shell.ts"; export type MuxBackendName = "tmux" | "zellij"; @@ -15,9 +15,71 @@ export type MuxSessionCreateResult = | { readonly ok: true; readonly session: MuxSession | null } | { readonly ok: false; readonly error: string; readonly stderr?: string }; +/** + * Bound for every non-interactive mux command. Some run while the shared + * helper instance lock is held (session decision and creation), so a hung + * tmux or zellij must not hold that lock indefinitely. A command that exceeds + * it has its client killed and is reported as {@link MuxCommandTimeoutError}: + * its outcome is unknown, never an empty or absent result. + * + * Killing the client bounds the caller's wait, not the command: a tmux server + * that already received it may still apply it later (commands queue per + * client), and zellij's background create daemonizes its session server. + * Callers therefore never treat a timeout as "did not happen"; the remaining + * late-effect residual is described in tests/models/tla/README.md ("Shared + * helper instance") and tracked in HACK-1849. `HACK_MUX_COMMAND_TIMEOUT_MS` overrides it (tests). + */ +export const MUX_COMMAND_TIMEOUT_MS = 15_000; + +export function muxCommandTimeoutMs(): number { + const override = Number(process.env.HACK_MUX_COMMAND_TIMEOUT_MS); + return Number.isInteger(override) && override > 0 + ? override + : MUX_COMMAND_TIMEOUT_MS; +} + +/** A mux command timed out and was killed: its outcome is unknown. */ +export class MuxCommandTimeoutError extends Error { + constructor(command: readonly string[]) { + super( + `${command.slice(0, 2).join(" ")} did not finish in time and was stopped; its outcome is unknown.` + ); + this.name = "MuxCommandTimeoutError"; + } +} + +/** + * Run a non-interactive mux command under {@link muxCommandTimeoutMs}. The + * call always returns within the bound; a timeout throws + * {@link MuxCommandTimeoutError} (the server may still apply the command). + */ +export async function runMuxCommand( + command: readonly string[], + opts: Omit & { + readonly timeoutMs?: number; + } = {} +): Promise { + const result = await exec(command, { + ...opts, + stdin: "ignore", + timeoutMs: opts.timeoutMs ?? muxCommandTimeoutMs(), + }); + if (result.exitCode === 124) { + throw new MuxCommandTimeoutError(command); + } + return result; +} + export interface MuxBackend { readonly name: MuxBackendName; readonly available: boolean; + /** + * The session carries its lifecycle ownership token from the creating + * command itself (tmux `new-session -e`), so no interrupted create can + * leave a live session without one. Shared-helper lifecycle sessions + * require it; zellij writes its token afterwards and lacks it (HACK-1849). + */ + readonly createsWithOwnerToken?: boolean; listSessions(): Promise; createSession(opts: { diff --git a/src/mux/tmux-backend.ts b/src/mux/tmux-backend.ts index 9f4195f50..78a68566c 100644 --- a/src/mux/tmux-backend.ts +++ b/src/mux/tmux-backend.ts @@ -1,10 +1,12 @@ import type { ExecResult, RunOptions } from "../lib/shell.ts"; -import { exec, findExecutableInPath } from "../lib/shell.ts"; +import { findExecutableInPath } from "../lib/shell.ts"; -import type { - MuxBackend, - MuxSession, - MuxSessionCreateResult, +import { + type MuxBackend, + MuxCommandTimeoutError, + type MuxSession, + type MuxSessionCreateResult, + runMuxCommand, } from "./mux-backend.ts"; const LIFECYCLE_OWNER_OPTION = "@hack_lifecycle_owner"; @@ -120,9 +122,12 @@ export function createTmuxBackend(): MuxBackend { // unset or C), tmux replaces control characters in client output with // `_`, which turns every line into one unmatched name: lifecycle // sessions then look absent to both `up` and `down` (HACK-1797). - const result = await exec(["tmux", "-u", "list-sessions", "-F", format], { - stdin: "ignore", - }); + const result = await runMuxCommand( + ["tmux", "-u", "list-sessions", "-F", format], + { + stdin: "ignore", + } + ); if (result.exitCode !== 0) { return []; @@ -156,7 +161,9 @@ export function createTmuxBackend(): MuxBackend { args.push("-e", `${LIFECYCLE_OWNER_ENV}=${opts.lifecycleOwnerToken}`); } - const result = await exec(args, { stdin: "ignore" }); + const result = await runMuxCommand(args, { + stdin: "ignore", + }); if (result.exitCode !== 0) { return { ok: false, @@ -166,7 +173,7 @@ export function createTmuxBackend(): MuxBackend { } if (opts.lifecycleOwnerToken) { - const ownerResult = await exec( + const ownerResult = await runMuxCommand( [ "tmux", "set-option", @@ -175,10 +182,10 @@ export function createTmuxBackend(): MuxBackend { LIFECYCLE_OWNER_OPTION, opts.lifecycleOwnerToken, ], - { stdin: "ignore" } + {} ); if (ownerResult.exitCode !== 0) { - await exec(["tmux", "kill-session", "-t", `=${opts.name}`], { + await runMuxCommand(["tmux", "kill-session", "-t", `=${opts.name}`], { stdin: "ignore", }); return { @@ -200,27 +207,41 @@ export function createTmuxBackend(): MuxBackend { // `=` makes tmux match the name exactly. A bare target falls back to a // prefix match, so stopping `demo` after it exited could kill // `demo--lifecycle`, a session the caller never verified. - return await exec(["tmux", "kill-session", "-t", `=${opts.name}`], { - stdin: "ignore", - }); + return await runMuxCommand( + ["tmux", "kill-session", "-t", `=${opts.name}`], + { + stdin: "ignore", + } + ); }; const readSessionPresence = async (opts: { readonly name: string }) => { if (!available) { return "unknown" as const; } - const result = await exec(["tmux", "has-session", "-t", `=${opts.name}`], { - stdin: "ignore", + const result = await runMuxCommand( + ["tmux", "has-session", "-t", `=${opts.name}`], + { + stdin: "ignore", + } + ).catch((error: unknown) => { + if (error instanceof MuxCommandTimeoutError) { + return null; + } + throw error; }); - return classifyTmuxSessionPresence({ result, name: opts.name }); + // A timed-out query proves nothing about presence. + return result === null + ? ("unknown" as const) + : classifyTmuxSessionPresence({ result, name: opts.name }); }; const readLifecycleOwnerToken = async (opts: { readonly name: string; }): Promise => { - const result = await exec( + const result = await runMuxCommand( ["tmux", "show-options", "-v", "-t", opts.name, LIFECYCLE_OWNER_OPTION], - { stdin: "ignore" } + {} ); if (result.exitCode === 0) { const token = result.stdout.trim(); @@ -228,9 +249,9 @@ export function createTmuxBackend(): MuxBackend { return token; } } - const environment = await exec( + const environment = await runMuxCommand( ["tmux", "show-environment", "-t", opts.name, LIFECYCLE_OWNER_ENV], - { stdin: "ignore" } + {} ); if (environment.exitCode !== 0) { return null; @@ -245,9 +266,9 @@ export function createTmuxBackend(): MuxBackend { const listSessionWindowNames = async (opts: { readonly name: string; }): Promise | null> => { - const result = await exec( + const result = await runMuxCommand( ["tmux", "list-windows", "-t", opts.name, "-F", "#{window_name}"], - { stdin: "ignore" } + {} ); if (result.exitCode !== 0) { return null; @@ -265,7 +286,7 @@ export function createTmuxBackend(): MuxBackend { readonly command: string; readonly env?: Readonly>; }): Promise => { - return await exec( + return await runMuxCommand( [ "tmux", "send-keys", @@ -287,14 +308,18 @@ export function createTmuxBackend(): MuxBackend { readonly name: string; readonly keys: string; }): Promise => { - return await exec(["tmux", "send-keys", "-t", opts.name, opts.keys], { - stdin: "ignore", - }); + return await runMuxCommand( + ["tmux", "send-keys", "-t", opts.name, opts.keys], + { + stdin: "ignore", + } + ); }; return { name: "tmux", available, + createsWithOwnerToken: true, listSessions, createSession, killSession, diff --git a/src/mux/zellij-backend.ts b/src/mux/zellij-backend.ts index 5136bf58b..dc7f52f15 100644 --- a/src/mux/zellij-backend.ts +++ b/src/mux/zellij-backend.ts @@ -1,10 +1,11 @@ import type { ExecResult, RunOptions } from "../lib/shell.ts"; -import { exec, findExecutableInPath } from "../lib/shell.ts"; +import { findExecutableInPath } from "../lib/shell.ts"; -import type { - MuxBackend, - MuxSession, - MuxSessionCreateResult, +import { + type MuxBackend, + type MuxSession, + type MuxSessionCreateResult, + runMuxCommand, } from "./mux-backend.ts"; const LIFECYCLE_OWNER_TAB_PREFIX = "hack-lifecycle-owner-"; @@ -20,7 +21,7 @@ export function createZellijBackend(): MuxBackend { return []; } - const result = await exec( + const result = await runMuxCommand( ["zellij", "list-sessions", "--no-formatting", "--short"], { stdin: "ignore", @@ -59,7 +60,7 @@ export function createZellijBackend(): MuxBackend { } // Create a detached session in the background. - const result = await exec( + const result = await runMuxCommand( ["zellij", "attach", "--create-background", opts.name], { stdin: "ignore", @@ -74,7 +75,7 @@ export function createZellijBackend(): MuxBackend { } if (opts.lifecycleOwnerToken) { - const ownerResult = await exec( + const ownerResult = await runMuxCommand( [ "zellij", "--session", @@ -88,7 +89,9 @@ export function createZellijBackend(): MuxBackend { } ); if (ownerResult.exitCode !== 0) { - await exec(["zellij", "kill-session", opts.name], { stdin: "ignore" }); + await runMuxCommand(["zellij", "kill-session", opts.name], { + stdin: "ignore", + }); return { ok: false, error: "owner_metadata_failed", @@ -105,7 +108,7 @@ export function createZellijBackend(): MuxBackend { const killSession = async (opts: { readonly name: string; }): Promise => { - return await exec(["zellij", "kill-session", opts.name], { + return await runMuxCommand(["zellij", "kill-session", opts.name], { stdin: "ignore", }); }; @@ -113,9 +116,9 @@ export function createZellijBackend(): MuxBackend { const readLifecycleOwnerToken = async (opts: { readonly name: string; }): Promise => { - const result = await exec( + const result = await runMuxCommand( ["zellij", "--session", opts.name, "action", "dump-layout"], - { stdin: "ignore" } + {} ); if (result.exitCode !== 0) { return null; @@ -127,9 +130,9 @@ export function createZellijBackend(): MuxBackend { const listSessionWindowNames = async (opts: { readonly name: string; }): Promise | null> => { - const result = await exec( + const result = await runMuxCommand( ["zellij", "--session", opts.name, "action", "dump-layout"], - { stdin: "ignore" } + {} ); if (result.exitCode !== 0) { return null; @@ -146,7 +149,7 @@ export function createZellijBackend(): MuxBackend { readonly command: string; readonly env?: Readonly>; }): Promise => { - return await exec( + return await runMuxCommand( [ "zellij", "--session", @@ -169,9 +172,9 @@ export function createZellijBackend(): MuxBackend { readonly keys: string; }): Promise => { // Best-effort. This sends raw characters and does not attempt to encode special key chords. - return await exec( + return await runMuxCommand( ["zellij", "--session", opts.name, "action", "write-chars", opts.keys], - { stdin: "ignore" } + {} ); }; diff --git a/src/templates.ts b/src/templates.ts index 8bdb81d5e..875e4051e 100644 --- a/src/templates.ts +++ b/src/templates.ts @@ -816,6 +816,51 @@ export function renderProjectConfigSchemaJson(): string { }, }, }, + sharedHelpers: { + type: "array", + items: { + type: "object", + additionalProperties: false, + required: ["name", "command", "port"], + oneOf: [ + { required: ["upstream"] }, + { required: ["upstreamEnv"] }, + ], + properties: { + name: { + type: "string", + pattern: "^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}$", + }, + enabled: { type: "boolean" }, + command: { + type: "array", + minItems: 1, + items: { type: "string", minLength: 1 }, + }, + cwd: { type: "string", minLength: 1 }, + port: { type: "integer", minimum: 1, maximum: 65_535 }, + identityPath: { type: "string", pattern: "^/" }, + upstream: { type: "string", minLength: 1 }, + upstreamEnv: { + type: "string", + pattern: "^[A-Za-z_][A-Za-z0-9_]*$", + }, + env: { + type: "array", + uniqueItems: true, + items: { + type: "string", + pattern: "^[A-Za-z_][A-Za-z0-9_]*$", + }, + }, + readyTimeoutMs: { + type: "integer", + minimum: 1, + maximum: 600_000, + }, + }, + }, + }, }, }, controlPlane: { diff --git a/tests/e2e/run.ts b/tests/e2e/run.ts index cf172c4a4..d13ffaaa4 100644 --- a/tests/e2e/run.ts +++ b/tests/e2e/run.ts @@ -68,6 +68,7 @@ import { patternP02PortableToolchainScenario } from "./scenarios/pattern-p02-por import { patternP07CapabilityOverlaysScenario } from "./scenarios/pattern-p07-capability-overlays.ts"; import { portableBootstrapNegativesScenario } from "./scenarios/portable-bootstrap-negatives.ts"; import { portableMultiserviceScenario } from "./scenarios/portable-multiservice.ts"; +import { sharedHostHelpersScenario } from "./scenarios/shared-host-helpers.ts"; import { upDownScenario } from "./scenarios/up-down.ts"; import { worktreeBranchDefaultScenario } from "./scenarios/worktree-branch-default.ts"; import { worktreeParallelUpScenario } from "./scenarios/worktree-parallel-up.ts"; @@ -109,6 +110,7 @@ export const ALL_SCENARIOS: readonly Scenario[] = [ doctorScenario, domainMigrationFilesScenario, lifecycleSessionRecoveryScenario, + sharedHostHelpersScenario, cachePruneScenario, dependencyCacheScenario, dependencyCacheBootstrapScenario, diff --git a/tests/e2e/scenarios/lifecycle-session-recovery.ts b/tests/e2e/scenarios/lifecycle-session-recovery.ts index 0446d1a71..5c9a741e4 100644 --- a/tests/e2e/scenarios/lifecycle-session-recovery.ts +++ b/tests/e2e/scenarios/lifecycle-session-recovery.ts @@ -672,7 +672,7 @@ function isProcessGroupAlive(processGroupId: number): boolean { } } -async function writeFakeDocker(opts: { +export async function writeFakeDocker(opts: { readonly fakeBin: string; }): Promise { await mkdir(opts.fakeBin, { recursive: true }); @@ -685,16 +685,29 @@ async function writeFakeDocker(opts: { 'if [ "$1" = "compose" ]; then', ' case " $* " in', ' *" ps "*)', + // A failed up started nothing, unless it failed after starting some + // services (E2E_DOCKER_FAIL_UP_PARTIAL). + ' if [ "${E2E_DOCKER_FAIL_UP:-}" = "1" ]; then exit 0; fi', ' printf \'%s\\n\' \'{"Service":"api","State":"running","ExitCode":0}\'', " exit 0", " ;;", ' *" up "*)', + // Optional gate: the image phase (`up --no-start`) records that it was + // reached, then waits until the gate file exists. + ' if [ -n "${E2E_DOCKER_GATE_UP:-}" ]; then', + ' case " $* " in *" --no-start "*)', + ' : > "${E2E_DOCKER_GATE_UP}.reached"', + ' while [ ! -e "$E2E_DOCKER_GATE_UP" ]; do sleep 0.05; done', + " ;;", + " esac", + " fi", ' if [ "${E2E_DOCKER_BLOCK_UP:-}" = "1" ]; then', ' parent_pid="$PPID"', ' while kill -0 "$parent_pid" 2>/dev/null; do sleep 0.1; done', " exit 143", " fi", ' if [ "${E2E_DOCKER_FAIL_UP:-}" = "1" ]; then exit 42; fi', + ' if [ "${E2E_DOCKER_FAIL_UP_PARTIAL:-}" = "1" ]; then exit 42; fi', " exit 0", " ;;", " *) exit 0 ;;", diff --git a/tests/e2e/scenarios/shared-host-helpers.ts b/tests/e2e/scenarios/shared-host-helpers.ts new file mode 100644 index 000000000..878420a55 --- /dev/null +++ b/tests/e2e/scenarios/shared-host-helpers.ts @@ -0,0 +1,1488 @@ +import { readFile, writeFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; + +import { readProcessIdentities } from "../../../src/lib/process-tree.ts"; +import { withProjectsRegistryLock } from "../../../src/lib/projects-registry-lock.ts"; +import { + isSharedHostProcessRunning, + parseSharedHostClaim, +} from "../../../src/lib/shared-host-lifecycle.ts"; +import { createMonorepoFixture } from "../fixture.ts"; +import { + buildCliEnv, + CLI_ENTRYPOINT, + type CliResult, + expect, + expectExit, + resolveCliSpawnArgs, + type Scenario, +} from "../harness.ts"; +import { writeFakeDocker } from "./lifecycle-session-recovery.ts"; + +/** + * Two branches of one project share one `lifecycle.sharedHelpers` entry + * through the real CLI (`hack up`/`down --branch`), with a fake Docker and + * the portable helper fixture from tests/fixtures/shared-host-helper.ts. + * + * Every phase asserts actual endpoint behavior, that no foreign process is + * signalled, and that no owned helper outlives its last holder. Foreign + * processes are started and stopped by this scenario through its own handles. + */ + +const HELPER = resolve(import.meta.dir, "../../fixtures/shared-host-helper.ts"); +const HELPER_NAME = "issuer-bridge"; +const UPSTREAM = "https://issuer.shared.test"; +const OTHER_UPSTREAM = "https://issuer.sibling.test"; +const WAIT_MS = 15_000; +const SIGNAL_EXIT_BOUND_MS = 5000; + +type Identity = { readonly pid: number; readonly birth: string }; +type HelperMode = "identity" | "placeholder"; + +export const sharedHostHelpersScenario: Scenario = { + name: "shared-host-helpers", + tier: "local", + summary: + "two branches share an identity-verified host helper through up/down without Docker", + run: async (ctx) => { + const fixture = await createMonorepoFixture({ + parentDir: ctx.tempRoot, + withHackConfig: true, + lifecycle: { disableInternal: true }, + }); + const fakeBin = join(ctx.tempRoot, "fake-bin"); + await writeFakeDocker({ fakeBin }); + const path = `${fakeBin}:${process.env.PATH ?? ""}`; + const port = freePort(); + const configPath = join(fixture.hackDir, "hack.config.json"); + const baseConfig = JSON.parse(await readFile(configPath, "utf8")) as Record< + string, + unknown + >; + const claimPath = join( + ctx.hackHome, + "shared-helpers", + `${HELPER_NAME}-${port}`, + `${HELPER_NAME}.claim.json` + ); + const branchA = "feat-a"; + const branchB = "feat-b"; + const holderA = `${fixture.name}--${branchA}`; + const holderB = `${fixture.name}--${branchB}`; + const foreign: Bun.Subprocess[] = []; + + const writeConfig = async (opts: { + readonly mode: HelperMode; + readonly readyTimeoutMs?: number; + /** Write the config with no sharedHelpers entry at all. */ + readonly withoutHelpers?: boolean; + /** Add a long-running lifecycle process (needs tmux). */ + readonly withProcess?: boolean; + /** Add a second entry with a shell-string command (invalid). */ + readonly withInvalidEntry?: boolean; + /** Write the helper entry with `enabled: false`. */ + readonly disabled?: boolean; + /** Add an up.before hook that waits for this gate file. */ + readonly beforeGate?: string; + /** Add an up.after hook that fails. */ + readonly failUpAfter?: boolean; + }): Promise => { + const lifecycle = { + ...((baseConfig.lifecycle ?? {}) as Record), + ...(opts.beforeGate + ? { + up: { + before: [ + { + name: "e2e-before-gate", + command: `: > "${opts.beforeGate}.reached"; while [ ! -e "${opts.beforeGate}" ]; do sleep 0.05; done`, + }, + ], + }, + } + : {}), + ...(opts.failUpAfter + ? { up: { after: [{ name: "e2e-after-fails", command: "exit 3" }] } } + : {}), + ...(opts.withProcess + ? { + processes: [ + { + name: "e2e-helper-client", + command: `${process.execPath} -e "setInterval(() => undefined, 60000)"`, + }, + ], + } + : {}), + }; + const helper = { + name: HELPER_NAME, + // HOME is the scenario temp root: the fixture exits once it is + // removed, so a failed run cannot leak a helper. + command: [ + "/usr/bin/env", + `HOME=${ctx.tempRoot}`, + process.execPath, + HELPER, + opts.mode, + String(port), + HELPER_NAME, + UPSTREAM, + ], + port, + upstreamEnv: "E2E_ISSUER_UPSTREAM", + readyTimeoutMs: opts.readyTimeoutMs ?? 10_000, + ...(opts.disabled ? { enabled: false } : {}), + }; + const sharedHelpers = opts.withInvalidEntry + ? [helper, { ...helper, name: "second-bridge", command: "bun bridge" }] + : [helper]; + await writeFile( + configPath, + `${JSON.stringify( + { + ...baseConfig, + lifecycle: opts.withoutHelpers + ? lifecycle + : { ...lifecycle, sharedHelpers }, + }, + null, + 2 + )}\n` + ); + }; + + const cli = (opts: { + readonly verb: "up" | "down" | "restart"; + readonly branch: string; + readonly upstream?: string; + readonly failCompose?: boolean; + /** Compose fails after starting some services. */ + readonly failComposePartial?: boolean; + }): Promise => + ctx.cli({ + args: [ + opts.verb, + ...(opts.verb === "up" ? ["--detach"] : []), + "--json", + "--branch", + opts.branch, + ], + cwd: fixture.root, + env: { + PATH: path, + E2E_ISSUER_UPSTREAM: opts.upstream ?? UPSTREAM, + ...(opts.failCompose ? { E2E_DOCKER_FAIL_UP: "1" } : {}), + ...(opts.failComposePartial + ? { E2E_DOCKER_FAIL_UP_PARTIAL: "1" } + : {}), + }, + }); + + const readClaim = async () => { + const text = await readFile(claimPath, "utf8").catch(() => null); + return text === null ? null : parseSharedHostClaim(text); + }; + const ownedHelper = async (): Promise => { + const claim = await readClaim(); + if (claim?.ownership !== "owned") { + throw new Error( + `expected an owned claim, got ${JSON.stringify(claim)}` + ); + } + return { pid: claim.pid, birth: claim.birth }; + }; + const startForeign = async ( + mode: "identity" | "tcp" | "placeholder" + ): Promise<{ readonly proc: Bun.Subprocess; readonly id: Identity }> => { + const proc = Bun.spawn( + [process.execPath, HELPER, mode, String(port), HELPER_NAME, UPSTREAM], + { + env: { PATH: process.env.PATH ?? "", HOME: ctx.tempRoot }, + stdin: "ignore", + stdout: "pipe", + stderr: "ignore", + } + ); + foreign.push(proc); + await waitForLine(proc.stdout, "listening"); + const id = await identityOf(proc.pid); + if (!id) { + throw new Error("foreign fixture missing from the census"); + } + return { proc, id }; + }; + const stopForeign = async (proc: Bun.Subprocess): Promise => { + proc.kill("SIGKILL"); + await proc.exited; + }; + + try { + await writeConfig({ mode: "identity" }); + + ctx.log("concurrent startup: two branches, one spawn"); + const [upA, upB] = await Promise.all([ + cli({ verb: "up", branch: branchA }), + cli({ verb: "up", branch: branchB }), + ]); + expectExit({ result: upA, codes: [0], message: "branch A up" }); + expectExit({ result: upB, codes: [0], message: "branch B up" }); + const shared = await ownedHelper(); + const claim = await readClaim(); + expect({ + that: + claim?.ownership === "owned" && + claim.phase === "ready" && + [...claim.holders].sort().join(",") === + [holderA, holderB].sort().join(","), + message: `both branches should hold one owned helper: ${JSON.stringify(claim)}`, + }); + expect({ + that: (await served(port)) === `served:${UPSTREAM}`, + message: "the shared helper endpoint should serve the upstream", + }); + + ctx.log("sibling branch with another upstream is refused"); + const sibling = await cli({ + verb: "up", + branch: "feat-c", + upstream: OTHER_UPSTREAM, + }); + expect({ + that: + sibling.exitCode !== 0 && + sibling.stdout.includes("E_LIFECYCLE_FAILED") && + sibling.combined.includes("wrong-upstream") && + !sibling.combined.includes(OTHER_UPSTREAM), + message: "a wrong-upstream sibling must fail closed without the value", + result: sibling, + }); + expect({ + that: + (await isRunning(shared)) && + (await readClaim())?.holders.length === 2, + message: "the refused sibling changed the shared helper", + result: sibling, + }); + + ctx.log("first down retains, last down stops"); + expectExit({ + result: await cli({ verb: "down", branch: branchA }), + codes: [0], + message: "branch A down", + }); + expect({ + that: + (await isRunning(shared)) && + (await served(port)) === `served:${UPSTREAM}` && + (await readClaim())?.holders.join(",") === holderB, + message: "branch A down must leave the helper for branch B", + }); + expectExit({ + result: await cli({ verb: "down", branch: branchB }), + codes: [0], + message: "branch B down", + }); + expect({ + that: !(await isRunning(shared)) && (await readClaim()) === null, + message: "the last down must stop the owned helper and drop its claim", + }); + + ctx.log("a failed up releases only the holder it added"); + expectExit({ + result: await cli({ verb: "up", branch: branchB }), + codes: [0], + message: "branch B up before A's failed up", + }); + const held = await ownedHelper(); + const failedUp = await cli({ + verb: "up", + branch: branchA, + failCompose: true, + }); + expect({ + that: + failedUp.exitCode !== 0 && + (await isRunning(held)) && + (await readClaim())?.holders.join(",") === holderB, + message: "a compose failure must roll back A's new holder, not B's", + result: failedUp, + }); + expectExit({ + result: await cli({ verb: "down", branch: branchB }), + codes: [0], + message: "branch B down after A's failed up", + }); + expect({ + that: !(await isRunning(held)) && (await readClaim()) === null, + message: "the last down after a failed up left the helper running", + }); + + ctx.log("child restart: a crashed owned helper is replaced"); + expectExit({ + result: await cli({ verb: "up", branch: branchA }), + codes: [0], + message: "branch A up before crash", + }); + const crashed = await ownedHelper(); + process.kill(crashed.pid, "SIGKILL"); + await waitUntil(async () => !(await isRunning(crashed))); + expectExit({ + result: await cli({ verb: "up", branch: branchA }), + codes: [0], + message: "branch A up after crash", + }); + const replaced = await ownedHelper(); + expect({ + that: + replaced.pid !== crashed.pid && + (await served(port)) === `served:${UPSTREAM}` && + (await readClaim())?.holders.join(",") === holderA, + message: "the crashed helper should be replaced under a fresh claim", + }); + expectExit({ + result: await cli({ verb: "down", branch: branchA }), + codes: [0], + message: "branch A down after restart", + }); + expect({ + that: !(await isRunning(replaced)), + message: "down left the replaced helper running", + }); + + ctx.log("stale PID: a reused PID is discarded and never signalled"); + const bystander = Bun.spawn( + [process.execPath, HELPER, "placeholder", "0", "-", "-"], + { + env: { PATH: process.env.PATH ?? "", HOME: ctx.tempRoot }, + stdin: "ignore", + stdout: "ignore", + stderr: "ignore", + } + ); + foreign.push(bystander); + await waitUntil(async () => (await identityOf(bystander.pid)) !== null); + await writeFile( + claimPath, + `${JSON.stringify({ + version: 1, + helper: HELPER_NAME, + port, + ownership: "owned", + phase: "ready", + pid: bystander.pid, + birth: "Thu Jan 1 00:00:00 1970", + holders: [holderB], + })}\n`, + { mode: 0o600 } + ); + const bystanderId = await identityOf(bystander.pid); + expectExit({ + result: await cli({ verb: "up", branch: branchA }), + codes: [0], + message: "up over a stale claim", + }); + const fresh = await ownedHelper(); + expect({ + that: + fresh.pid !== bystander.pid && + bystanderId !== null && + (await isRunning(bystanderId)) && + (await readClaim())?.holders.includes(holderA) === true, + message: "a stale claim must be discarded without signalling its PID", + }); + // Drop the carried stale holder, then A's release stops the helper. + expectExit({ + result: await cli({ verb: "down", branch: branchB }), + codes: [0], + message: "branch B down (carried holder)", + }); + expectExit({ + result: await cli({ verb: "down", branch: branchA }), + codes: [0], + message: "branch A down after stale claim", + }); + expect({ + that: + !(await isRunning(fresh)) && + bystanderId !== null && + (await isRunning(bystanderId)), + message: "down must stop only the owned helper", + }); + await stopForeign(bystander); + + ctx.log("healthy helper started elsewhere is adopted, never stopped"); + const external = await startForeign("identity"); + expectExit({ + result: await cli({ verb: "up", branch: branchA }), + codes: [0], + message: "branch A adopts an external helper", + }); + expect({ + that: (await readClaim())?.ownership === "adopted", + message: "the external helper should be recorded as adopted", + }); + expectExit({ + result: await cli({ verb: "down", branch: branchA }), + codes: [0], + message: "branch A down after adoption", + }); + expect({ + that: + (await isRunning(external.id)) && + (await served(port)) === `served:${UPSTREAM}` && + (await readClaim()) === null, + message: "down must release, not stop, an adopted helper", + }); + await stopForeign(external.proc); + + ctx.log("foreign port owner is refused and survives"); + const tcp = await startForeign("tcp"); + const refused = await cli({ verb: "up", branch: branchA }); + expect({ + that: + refused.exitCode !== 0 && + refused.combined.includes("foreign-listener") && + (await isRunning(tcp.id)) && + (await readClaim()) === null, + message: "a foreign listener must fail closed and keep running", + result: refused, + }); + await stopForeign(tcp.proc); + await waitUntil(async () => !(await portBusy(port))); + + ctx.log("shutdown during readiness retry leaves no owned orphan"); + await writeConfig({ mode: "placeholder", readyTimeoutMs: 30_000 }); + const term = await signalDuringRetry({ + signal: "SIGTERM", + cwd: fixture.root, + hackHome: ctx.hackHome, + path, + branch: branchA, + readClaim, + }); + expect({ + that: + term.exitCode === 143 && + // Promptly: a start that ignored the signal would only clean up at + // its 30 s readiness deadline. + term.exitMs < SIGNAL_EXIT_BOUND_MS && + !(await isRunning(term.helper)) && + (await readClaim()) === null, + message: `SIGTERM during retry must promptly stop the owned child and drop its claim (exit ${term.exitCode} after ${term.exitMs} ms)`, + }); + + ctx.log("lost parent: the next starter stops the abandoned helper"); + const lost = await signalDuringRetry({ + signal: "SIGKILL", + cwd: fixture.root, + hackHome: ctx.hackHome, + path, + branch: branchA, + readClaim, + }); + expect({ + that: + (await isRunning(lost.helper)) && + (await readClaim())?.ownership === "owned", + message: "a killed starter leaves its helper recorded, not unclaimed", + }); + await writeConfig({ mode: "placeholder", readyTimeoutMs: 1500 }); + const recovered = await cli({ verb: "up", branch: branchB }); + expect({ + that: + recovered.exitCode !== 0 && + recovered.combined.includes("deadline") && + !(await isRunning(lost.helper)) && + (await readClaim()) === null, + message: "the next starter must stop the abandoned helper", + result: recovered, + }); + await writeConfig({ mode: "identity" }); + expectExit({ + result: await cli({ verb: "up", branch: branchB }), + codes: [0], + message: "a later up starts fresh", + }); + const last = await ownedHelper(); + expectExit({ + result: await cli({ verb: "down", branch: branchB }), + codes: [0], + message: "final down", + }); + expect({ + that: !((await isRunning(last)) || (await portBusy(port))), + message: "the final down left an owned helper or listener behind", + }); + + await runReviewRegressionPhases({ + ctx, + cwd: fixture.root, + path, + branch: branchA, + claimPath, + writeConfig, + cli, + readClaim, + ownedHelper, + }); + await runSecondReviewPhases({ + ctx, + cwd: fixture.root, + hackDir: fixture.hackDir, + fixtureName: fixture.name, + path, + port, + branch: branchA, + writeConfig, + cli, + readClaim, + ownedHelper, + }); + await runRetainedHelperPhases({ + ctx, + cwd: fixture.root, + path, + port, + holder: holderA, + branch: branchA, + writeConfig, + cli, + readClaim, + ownedHelper, + }); + } finally { + await cleanupScenario({ foreign, readClaim }); + } + }, +}; + +/** + * Helpers stay held while their containers run: a restart that preserves + * the Compose runtime keeps them, and so does an up whose up.after hook + * fails after its detached containers started. + */ +async function runRetainedHelperPhases(opts: { + readonly ctx: Parameters[0]; + readonly cwd: string; + readonly path: string; + readonly port: number; + readonly holder: string; + readonly branch: string; + readonly writeConfig: (opts: ConfigOptions) => Promise; + readonly cli: (opts: { + readonly verb: "up" | "down" | "restart"; + readonly branch: string; + readonly failComposePartial?: boolean; + }) => Promise; + readonly readClaim: () => Promise | null>; + readonly ownedHelper: () => Promise; +}): Promise { + const { ctx, branch } = opts; + const keptFor = async (helper: Identity): Promise => + (await isRunning(helper)) && + (await served(opts.port)) === `served:${UPSTREAM}` && + (await opts.readClaim())?.holders.join(",") === opts.holder; + const downStops = async (helper: Identity, phase: string): Promise => { + expectExit({ + result: await opts.cli({ verb: "down", branch }), + codes: [0], + message: `down after ${phase}`, + }); + expect({ + that: !(await isRunning(helper)) && (await opts.readClaim()) === null, + message: `down after ${phase} left the helper running`, + }); + }; + + ctx.log("restart keeps the helper of the containers it preserves"); + await opts.writeConfig({ mode: "identity" }); + expectExit({ + result: await opts.cli({ verb: "up", branch }), + codes: [0], + message: "up before restart", + }); + const beforeRestart = await opts.ownedHelper(); + // Restart preserves the Compose runtime, so the containers keep running: + // their sole holder must not stop the helper underneath them. + const restarted = await opts.cli({ verb: "restart", branch }); + expectExit({ result: restarted, codes: [0], message: "restart" }); + const afterRestart = await opts.ownedHelper(); + expect({ + that: + afterRestart.pid === beforeRestart.pid && + afterRestart.birth === beforeRestart.birth && + (await keptFor(afterRestart)), + message: + "restart stopped or replaced the helper of its preserved containers", + result: restarted, + }); + await downStops(afterRestart, "restart"); + + ctx.log("a failing up.after hook keeps the helper of the started containers"); + await opts.writeConfig({ mode: "identity", failUpAfter: true }); + const failedAfter = await opts.cli({ verb: "up", branch }); + const claimAfter = await opts.readClaim(); + expect({ + that: + failedAfter.exitCode !== 0 && + claimAfter?.ownership === "owned" && + (await keptFor({ pid: claimAfter.pid, birth: claimAfter.birth })), + message: + "a failing up.after hook released the helper under running containers", + result: failedAfter, + }); + const started = await opts.ownedHelper(); + await opts.writeConfig({ mode: "identity" }); + await downStops(started, "a failed up.after hook"); + + ctx.log("a Compose failure after a partial start keeps the helper"); + // Compose exits non-zero (as on a dependency failure or a timeout) after + // starting some services, which `ps` still reports as running. + const partial = await opts.cli({ + verb: "up", + branch, + failComposePartial: true, + }); + const claimAfterPartial = await opts.readClaim(); + expect({ + that: + partial.exitCode !== 0 && + claimAfterPartial?.ownership === "owned" && + (await keptFor({ + pid: claimAfterPartial.pid, + birth: claimAfterPartial.birth, + })), + message: + "a Compose failure after a partial start released the helper under running services", + result: partial, + }); + await downStops(await opts.ownedHelper(), "a partial start"); + + ctx.log("a signal during a container start keeps the helper"); + // The up pauses inside docker compose up, with a service already reported + // running; SIGTERM reaches the CLI while that start is still in flight. + const gate = join(ctx.tempRoot, "signal-during-start-gate"); + const signalled = await startGatedUp({ + ctx, + cwd: opts.cwd, + path: opts.path, + branch, + gate, + }); + const beforeSignal = await opts.ownedHelper(); + signalled.kill("SIGTERM"); + await writeFile(gate, "open\n"); + const signalledExit = await signalled.exited; + expect({ + that: signalledExit !== 0 && (await keptFor(beforeSignal)), + message: `a signal during a container start released the helper under started services (exit ${signalledExit})`, + }); + await downStops(beforeSignal, "a signal during a container start"); +} + +/** + * Regressions from the independent review of #565: a down that completes + * while an up is paused after helper preparation; a helper entry removed + * from config after start; and a down whose helper release fails. + */ +async function runReviewRegressionPhases(opts: { + readonly ctx: Parameters[0]; + readonly cwd: string; + readonly path: string; + readonly branch: string; + readonly claimPath: string; + readonly writeConfig: (opts: ConfigOptions) => Promise; + readonly cli: (opts: { + readonly verb: "up" | "down"; + readonly branch: string; + }) => Promise; + readonly readClaim: () => Promise | null>; + readonly ownedHelper: () => Promise; +}): Promise { + const { ctx, branch } = opts; + await opts.writeConfig({ mode: "identity" }); + + ctx.log("down during a paused up: the up refuses to start services"); + const gate = join(ctx.tempRoot, "compose-up-gate"); + const dockerLog = join(ctx.tempRoot, "gated-docker.log"); + const pausedUp = Bun.spawn( + resolveCliSpawnArgs(["up", "--detach", "--json", "--branch", branch]), + { + cwd: opts.cwd, + env: buildCliEnv({ + hackHome: ctx.hackHome, + extra: { + PATH: opts.path, + E2E_ISSUER_UPSTREAM: UPSTREAM, + E2E_DOCKER_GATE_UP: gate, + E2E_DOCKER_LOG: dockerLog, + }, + }), + stdin: "ignore", + stdout: "pipe", + stderr: "ignore", + } + ); + let reached = false; + await waitUntil(async () => { + reached = await Bun.file(`${gate}.reached`).exists(); + return reached; + }); + if (!reached) { + pausedUp.kill("SIGKILL"); + await pausedUp.exited; + throw new Error("the gated up never reached compose image preparation"); + } + // Helper preparation is done: the up holds a ready helper. + const pausedHelper = await opts.ownedHelper(); + expectExit({ + result: await opts.cli({ verb: "down", branch }), + codes: [0], + message: "down while the up is paused", + }); + expect({ + that: !(await isRunning(pausedHelper)) && (await opts.readClaim()) === null, + message: "down must release the paused up's helper", + }); + await writeFile(gate, "open\n"); + const [pausedExit, pausedStdout] = await Promise.all([ + pausedUp.exited, + new Response(pausedUp.stdout).text(), + ]); + const pausedResult = parseEnvelope(pausedStdout); + const startCalls = (await Bun.file(dockerLog).text()) + .split("\n") + .filter((line) => line.includes(" up -d ")); + expect({ + that: + pausedExit !== 0 && + pausedResult?.error?.code === "E_LIFECYCLE_FAILED" && + (pausedResult.error.message ?? "").includes( + "before launching services" + ) && + startCalls.length === 0 && + (await opts.readClaim()) === null, + message: `a superseded up must fail before starting containers (exit ${pausedExit}, stdout ${pausedStdout.trim()}, start calls ${startCalls.length})`, + }); + + ctx.log("a helper entry removed after start is still released by down"); + expectExit({ + result: await opts.cli({ verb: "up", branch }), + codes: [0], + message: "up before removing the helper entry", + }); + const removedHelper = await opts.ownedHelper(); + await opts.writeConfig({ mode: "identity", withoutHelpers: true }); + expectExit({ + result: await opts.cli({ verb: "down", branch }), + codes: [0], + message: "down after removing the helper entry", + }); + expect({ + that: + !(await isRunning(removedHelper)) && (await opts.readClaim()) === null, + message: "down must release a helper whose config entry was removed", + }); + await opts.writeConfig({ mode: "identity" }); + + ctx.log("an unreleased helper fails down and keeps its claim"); + expectExit({ + result: await opts.cli({ verb: "up", branch }), + codes: [0], + message: "up before corrupting the claim", + }); + const keptHelper = await opts.ownedHelper(); + const claimText = await readFile(opts.claimPath, "utf8"); + await writeFile(opts.claimPath, "{not json\n", { mode: 0o600 }); + const failedDown = await opts.cli({ verb: "down", branch }); + const failedResult = parseEnvelope(failedDown.stdout); + expect({ + that: + failedDown.exitCode !== 0 && + failedResult?.ok === false && + failedResult.error?.code === "E_LIFECYCLE_FAILED" && + JSON.stringify(failedResult.error.detail ?? {}).includes(HELPER_NAME) && + (await readFile(opts.claimPath, "utf8")) === "{not json\n" && + (await isRunning(keptHelper)), + message: "a failed helper release must fail down and keep the claim", + result: failedDown, + }); + await writeFile(opts.claimPath, claimText, { mode: 0o600 }); + expectExit({ + result: await opts.cli({ verb: "down", branch }), + codes: [0], + message: "down after restoring the claim", + }); + expect({ + that: !(await isRunning(keptHelper)) && (await opts.readClaim()) === null, + message: "the retried down must stop the helper", + }); +} + +type ConfigOptions = { + readonly mode: HelperMode; + readonly withoutHelpers?: boolean; + readonly withProcess?: boolean; + readonly withInvalidEntry?: boolean; + readonly disabled?: boolean; + readonly beforeGate?: string; + readonly failUpAfter?: boolean; +}; + +/** + * Regressions from the second independent review of #565: an invalid entry + * is refused before any effect; a release failure with no helper configured + * still fails down; and a newer up of the same instance never makes an + * earlier, slower up tear down the lifecycle session it adopted. + */ +async function runSecondReviewPhases(opts: { + readonly ctx: Parameters[0]; + readonly cwd: string; + readonly hackDir: string; + readonly fixtureName: string; + readonly path: string; + readonly port: number; + readonly branch: string; + readonly writeConfig: (opts: ConfigOptions) => Promise; + readonly cli: (opts: { + readonly verb: "up" | "down"; + readonly branch: string; + }) => Promise; + readonly readClaim: () => Promise | null>; + readonly ownedHelper: () => Promise; +}): Promise { + const { ctx, branch } = opts; + const holder = `${opts.fixtureName}--${branch}`; + + ctx.log("an invalid sharedHelpers entry is refused before any effect"); + await opts.writeConfig({ mode: "identity", withInvalidEntry: true }); + const refusalLog = join(ctx.tempRoot, "refusal-docker.log"); + for (const verb of ["up", "down", "restart"] as const) { + const refused = await ctx.cli({ + args: [ + verb, + ...(verb === "up" ? ["--detach"] : []), + "--json", + "--branch", + branch, + ], + cwd: opts.cwd, + env: { + PATH: opts.path, + E2E_ISSUER_UPSTREAM: UPSTREAM, + E2E_DOCKER_LOG: refusalLog, + }, + }); + const envelope = parseEnvelope(refused.stdout); + expect({ + that: + refused.exitCode !== 0 && + envelope?.error?.code === "E_LIFECYCLE_FAILED" && + (envelope.error.message ?? "").includes( + "Invalid lifecycle.sharedHelpers" + ), + message: `hack ${verb} must refuse an invalid sharedHelpers entry`, + result: refused, + }); + } + expect({ + that: + !(await Bun.file(refusalLog).exists()) && + (await opts.readClaim()) === null && + !(await portBusy(opts.port)), + message: + "a refused command must run no Compose command and start no helper", + }); + await opts.writeConfig({ mode: "identity" }); + + ctx.log("a release failure with no helper configured still fails down"); + expectExit({ + result: await opts.cli({ verb: "up", branch }), + codes: [0], + message: "up before removing the entry and corrupting the record", + }); + const recordedHelper = await opts.ownedHelper(); + const acquiredPath = join( + ctx.hackHome, + "shared-helpers", + ".instances", + `${holder}.acquired.json` + ); + const acquiredText = await readFile(acquiredPath, "utf8"); + await opts.writeConfig({ mode: "identity", withoutHelpers: true }); + await writeFile(acquiredPath, "{not json\n", { mode: 0o600 }); + const unlistedDown = await opts.cli({ verb: "down", branch }); + const unlisted = parseEnvelope(unlistedDown.stdout); + expect({ + that: + unlistedDown.exitCode !== 0 && + unlisted?.error?.code === "E_LIFECYCLE_FAILED" && + JSON.stringify(unlisted.error.detail ?? {}).includes( + "recorded-helpers" + ) && + (await isRunning(recordedHelper)), + message: + "an unreadable acquired record must fail down even with no helper configured", + result: unlistedDown, + }); + await writeFile(acquiredPath, acquiredText, { mode: 0o600 }); + expectExit({ + result: await opts.cli({ verb: "down", branch }), + codes: [0], + message: "down after restoring the acquired record", + }); + expect({ + that: !(await isRunning(recordedHelper)), + message: "the retried down must stop the recorded helper", + }); + + await runOverlappingUpPhase(opts); + await runDisabledAfterDownPhase(opts); + await runDownDuringBeforeHookPhase(opts); + await opts.writeConfig({ mode: "identity" }); +} + +type OverlapOptions = { + readonly ctx: Parameters[0]; + readonly port: number; + readonly cwd: string; + readonly hackDir: string; + readonly fixtureName: string; + readonly path: string; + readonly branch: string; + readonly writeConfig: (opts: ConfigOptions) => Promise; + readonly cli: (opts: { + readonly verb: "up" | "down"; + readonly branch: string; + }) => Promise; + readonly ownedHelper: () => Promise; +}; + +/** + * Start `hack up` for the branch, paused at Compose image preparation by the + * fake Docker gate. The CLI process itself is spawned (not a `bun` wrapper) + * so a signal reaches Hack's own handlers. + */ +async function startGatedUp(opts: { + readonly ctx: Parameters[0]; + readonly cwd: string; + readonly path: string; + readonly branch: string; + readonly gate: string; + readonly extraEnv?: Readonly>; +}): Promise> { + const args = ["up", "--detach", "--json", "--branch", opts.branch]; + const proc = Bun.spawn( + process.env.HACK_E2E_CLI_BIN?.trim() + ? resolveCliSpawnArgs(args) + : [process.execPath, CLI_ENTRYPOINT, ...args], + { + cwd: opts.cwd, + env: buildCliEnv({ + hackHome: opts.ctx.hackHome, + extra: { + PATH: opts.path, + E2E_ISSUER_UPSTREAM: UPSTREAM, + E2E_DOCKER_GATE_UP: opts.gate, + ...opts.extraEnv, + }, + }), + stdin: "ignore", + stdout: "pipe", + stderr: "ignore", + } + ); + let reached = false; + await waitUntil(async () => { + reached = await Bun.file(`${opts.gate}.reached`).exists(); + return reached; + }); + if (!reached) { + proc.kill("SIGKILL"); + await proc.exited; + throw new Error("the gated up never reached compose image preparation"); + } + return proc; +} + +type EarlierOutcome = + | "succeeds" + | "compose-fails" + | "sigterm" + /** SIGTERM while another start holds the instance lock past the cleanup timeout. */ + | "sigterm-lock-held"; + +/** + * A newer up of the same instance adopts the lifecycle session and helpers + * of an earlier up that is paused at image preparation. Whatever the earlier + * up then does (succeeds, fails in Compose, or is terminated), it must not + * tear down the session or helpers the newer up now relies on. + */ +async function runOverlappingUpPhase(opts: OverlapOptions): Promise { + const { ctx } = opts; + const tmux = Bun.which("tmux"); + if (!tmux) { + if (process.env.HACK_E2E_REQUIRE_TMUX === "1") { + throw new Error("tmux is required for the overlapping-up phase"); + } + ctx.log("overlapping up phase skipped: tmux is not installed"); + return; + } + await opts.writeConfig({ mode: "identity", withProcess: true }); + for (const outcome of [ + "succeeds", + "compose-fails", + "sigterm", + "sigterm-lock-held", + ] as const satisfies readonly EarlierOutcome[]) { + await runOverlap({ ...opts, tmux, outcome }); + } +} + +async function runOverlap( + opts: OverlapOptions & { + readonly tmux: string; + readonly outcome: EarlierOutcome; + } +): Promise { + const { ctx, branch, outcome } = opts; + ctx.log( + `a newer up never makes an earlier up tear down its session (earlier up ${outcome})` + ); + const sessionName = `${opts.fixtureName}--lifecycle-${branch}`; + const composeProject = `${opts.fixtureName}--${branch}`; + const statePath = join(opts.hackDir, ".internal", "lifecycle", "state.json"); + const readToken = async (): Promise => { + const text = await readFile(statePath, "utf8").catch(() => null); + if (text === null) { + return null; + } + const state = JSON.parse(text) as { + readonly entries?: readonly { + readonly composeProject?: string; + readonly ownershipToken?: string; + }[]; + }; + return ( + state.entries?.find((entry) => entry.composeProject === composeProject) + ?.ownershipToken ?? null + ); + }; + const sessionAlive = async (): Promise => + (await Bun.spawn([opts.tmux, "has-session", "-t", sessionName], { + stdout: "ignore", + stderr: "ignore", + }).exited) === 0; + + const gate = join(ctx.tempRoot, `overlap-gate-${outcome}`); + const earlier = await startGatedUp({ + ctx, + cwd: opts.cwd, + path: opts.path, + branch, + gate, + ...(outcome === "compose-fails" + ? { extraEnv: { E2E_DOCKER_FAIL_UP: "1" } } + : {}), + }); + const helper = await opts.ownedHelper(); + const token = await readToken(); + expect({ + that: token !== null && (await sessionAlive()), + message: "the earlier up should own a lifecycle session before pausing", + }); + expectExit({ + result: await opts.cli({ verb: "up", branch }), + codes: [0], + message: "the newer up should adopt the session and helpers", + }); + // For sigterm-lock-held, hold the shared helper instance lock (as a third + // start ensuring a slow helper would) until the earlier up has exited. + const lock = + outcome === "sigterm-lock-held" + ? await holdInstanceLock( + join( + ctx.hackHome, + "shared-helpers", + ".instances", + `${composeProject}.lock` + ) + ) + : null; + if (outcome === "sigterm" || outcome === "sigterm-lock-held") { + earlier.kill("SIGTERM"); + } + await writeFile(gate, "open\n"); + const [earlierExit, earlierStdout] = await Promise.all([ + earlier.exited, + new Response(earlier.stdout).text(), + ]); + await lock?.release(); + const expectedExit = + outcome === "succeeds" ? earlierExit === 0 : earlierExit !== 0; + expect({ + that: + expectedExit && + (await sessionAlive()) && + (await readToken()) === token && + (await isRunning(helper)), + message: `the earlier up (${outcome}) must not tear down the adopted session or helpers (exit ${earlierExit}, stdout ${earlierStdout.trim().slice(0, 300)})`, + }); + expectExit({ + result: await opts.cli({ verb: "down", branch }), + codes: [0], + message: "down after the overlapping ups", + }); + expect({ + that: !((await sessionAlive()) || (await isRunning(helper))), + message: "down must stop the session and the helper", + }); +} + +/** Hold a Hack lock file until `release` is called. */ +async function holdInstanceLock( + lockPath: string +): Promise<{ readonly release: () => Promise }> { + let releaseLock = (): void => undefined; + const held = new Promise((resolveHeld) => { + releaseLock = resolveHeld; + }); + let acquired = (): void => undefined; + const isHeld = new Promise((resolveAcquired) => { + acquired = resolveAcquired; + }); + const holder = withProjectsRegistryLock({ + lockPath, + run: async () => { + acquired(); + await held; + }, + }); + await isHeld; + return { + release: async () => { + releaseLock(); + await holder; + }, + }; +} + +/** + * A down while an up waits in an up.before hook: the shutdown generation is + * captured before the hooks, so that up must refuse to start helpers or + * services once the hook finishes. + */ +async function runDownDuringBeforeHookPhase( + opts: OverlapOptions +): Promise { + const { ctx, branch } = opts; + ctx.log("a down while an up waits in up.before supersedes that up"); + const gate = join(ctx.tempRoot, "before-hook-gate"); + const dockerLog = join(ctx.tempRoot, "before-hook-docker.log"); + await opts.writeConfig({ mode: "identity", beforeGate: gate }); + const args = ["up", "--detach", "--json", "--branch", branch]; + const earlier = Bun.spawn( + process.env.HACK_E2E_CLI_BIN?.trim() + ? resolveCliSpawnArgs(args) + : [process.execPath, CLI_ENTRYPOINT, ...args], + { + cwd: opts.cwd, + env: buildCliEnv({ + hackHome: ctx.hackHome, + extra: { + PATH: opts.path, + E2E_ISSUER_UPSTREAM: UPSTREAM, + E2E_DOCKER_LOG: dockerLog, + }, + }), + stdin: "ignore", + stdout: "pipe", + stderr: "ignore", + } + ); + let reached = false; + await waitUntil(async () => { + reached = await Bun.file(`${gate}.reached`).exists(); + return reached; + }); + if (!reached) { + earlier.kill("SIGKILL"); + await earlier.exited; + throw new Error("the up never reached its up.before gate"); + } + expectExit({ + result: await opts.cli({ verb: "down", branch }), + codes: [0], + message: "down while the up waits in up.before", + }); + await writeFile(gate, "open\n"); + const [earlierExit, earlierStdout] = await Promise.all([ + earlier.exited, + new Response(earlier.stdout).text(), + ]); + const envelope = parseEnvelope(earlierStdout); + const upCalls = (await Bun.file(dockerLog).exists()) + ? (await Bun.file(dockerLog).text()) + .split("\n") + .filter((line) => line.includes(" up ")) + : []; + expect({ + that: + earlierExit !== 0 && + envelope?.error?.code === "E_LIFECYCLE_FAILED" && + upCalls.length === 0 && + !(await portBusy(opts.port)), + message: `an up superseded during up.before must start no helper or service (exit ${earlierExit}, compose up calls ${upCalls.length}, stdout ${earlierStdout.trim().slice(0, 300)})`, + }); + await opts.writeConfig({ mode: "identity" }); +} + +/** + * A down while an up is paused, then a newer up that succeeds with the helper + * disabled: the newer start does not restore the earlier start's helper, so + * the earlier up must still refuse to start services. + */ +async function runDisabledAfterDownPhase(opts: OverlapOptions): Promise { + const { ctx, branch } = opts; + ctx.log( + "a newer up with the helper disabled does not restore an earlier up after down" + ); + await opts.writeConfig({ mode: "identity" }); + const gate = join(ctx.tempRoot, "disabled-gate"); + const dockerLog = join(ctx.tempRoot, "disabled-docker.log"); + const earlier = await startGatedUp({ + ctx, + cwd: opts.cwd, + path: opts.path, + branch, + gate, + extraEnv: { E2E_DOCKER_LOG: dockerLog }, + }); + const helper = await opts.ownedHelper(); + expectExit({ + result: await opts.cli({ verb: "down", branch }), + codes: [0], + message: "down while the earlier up is paused", + }); + await opts.writeConfig({ mode: "identity", disabled: true }); + expectExit({ + result: await opts.cli({ verb: "up", branch }), + codes: [0], + message: "the newer up with the helper disabled", + }); + await writeFile(gate, "open\n"); + const [earlierExit, earlierStdout] = await Promise.all([ + earlier.exited, + new Response(earlier.stdout).text(), + ]); + const envelope = parseEnvelope(earlierStdout); + const startCalls = (await Bun.file(dockerLog).text()) + .split("\n") + .filter((line) => line.includes(" up -d ")); + expect({ + that: + earlierExit !== 0 && + envelope?.error?.code === "E_LIFECYCLE_FAILED" && + startCalls.length === 0 && + !(await isRunning(helper)), + message: `the earlier up must stay superseded after a later up with the helper disabled (exit ${earlierExit}, start calls ${startCalls.length})`, + }); + expectExit({ + result: await opts.cli({ verb: "down", branch }), + codes: [0], + message: "down after the disabled-helper up", + }); + await opts.writeConfig({ mode: "identity" }); +} + +type Envelope = { + readonly ok?: boolean; + readonly error?: { + readonly code?: string; + readonly message?: string; + readonly detail?: unknown; + }; +}; + +function parseEnvelope(stdout: string): Envelope | null { + // The CLI prints one (pretty-printed) JSON envelope on stdout. + const start = stdout.indexOf("{"); + if (start < 0) { + return null; + } + try { + return JSON.parse(stdout.slice(start)) as Envelope; + } catch { + return null; + } +} + +/** Kill this scenario's foreign fixtures and any helper still claimed as owned. */ +async function cleanupScenario(opts: { + readonly foreign: readonly Bun.Subprocess[]; + readonly readClaim: () => Promise | null>; +}): Promise { + for (const proc of opts.foreign) { + proc.kill("SIGKILL"); + await proc.exited; + } + const claim = await opts.readClaim(); + if (claim?.ownership === "owned") { + const id = { pid: claim.pid, birth: claim.birth }; + if (await isRunning(id)) { + process.kill(id.pid, "SIGKILL"); + } + } +} + +async function signalDuringRetry(opts: { + readonly signal: "SIGTERM" | "SIGKILL"; + readonly cwd: string; + readonly hackHome: string; + readonly path: string; + readonly branch: string; + readonly readClaim: () => Promise | null>; +}): Promise<{ + readonly exitCode: number; + readonly helper: Identity; + /** Time from the signal to the CLI's exit. */ + readonly exitMs: number; +}> { + // Signal the CLI process itself: a `bun` on PATH may be a wrapper that + // forwards or widens the signal (for example by stopping its whole process + // scope), which would hide whether Hack's own handler cleaned up. + const args = ["up", "--detach", "--json", "--branch", opts.branch]; + const argv = process.env.HACK_E2E_CLI_BIN?.trim() + ? resolveCliSpawnArgs(args) + : [process.execPath, CLI_ENTRYPOINT, ...args]; + const proc = Bun.spawn(argv, { + cwd: opts.cwd, + env: buildCliEnv({ + hackHome: opts.hackHome, + extra: { PATH: opts.path, E2E_ISSUER_UPSTREAM: UPSTREAM }, + }), + stdin: "ignore", + stdout: "ignore", + stderr: "ignore", + }); + let helper: Identity | null = null; + const deadline = Date.now() + WAIT_MS; + while (helper === null && Date.now() < deadline) { + const claim = await opts.readClaim(); + if (claim?.ownership === "owned" && claim.phase === "starting") { + helper = { pid: claim.pid, birth: claim.birth }; + } + await Bun.sleep(50); + } + if (helper === null) { + proc.kill("SIGKILL"); + await proc.exited; + throw new Error("the placeholder helper was never claimed"); + } + // Let readiness retries begin after the gate opened. + await Bun.sleep(500); + const signalledAt = Date.now(); + proc.kill(opts.signal); + const exitCode = await proc.exited; + return { exitCode, helper, exitMs: Date.now() - signalledAt }; +} + +function freePort(): number { + const probe = Bun.listen({ + hostname: "127.0.0.1", + port: 0, + socket: { + data() { + // Bind probe only. + }, + }, + }); + const port = probe.port; + probe.stop(true); + return port; +} + +function portBusy(port: number): Promise { + try { + const probe = Bun.listen({ + hostname: "127.0.0.1", + port, + socket: { + data() { + // Bind probe only. + }, + }, + }); + probe.stop(true); + return Promise.resolve(false); + } catch { + return Promise.resolve(true); + } +} + +async function served(port: number): Promise { + try { + return await ( + await fetch(`http://127.0.0.1:${port}/`, { + signal: AbortSignal.timeout(2000), + }) + ).text(); + } catch { + return ""; + } +} + +async function identityOf(pid: number): Promise { + const row = (await readProcessIdentities()).find( + (entry) => entry.pid === pid + ); + return row ? { pid: row.pid, birth: row.birth } : null; +} + +async function isRunning(identity: Identity): Promise { + return isSharedHostProcessRunning(await readProcessIdentities(), identity); +} + +async function waitUntil(check: () => Promise): Promise { + const deadline = Date.now() + WAIT_MS; + while (Date.now() < deadline) { + if (await check()) { + return; + } + await Bun.sleep(50); + } +} + +async function waitForLine( + stream: ReadableStream | number | undefined, + line: string +): Promise { + if (!(stream && typeof stream !== "number")) { + throw new Error("fixture stdout is not piped"); + } + const reader = stream.getReader(); + const decoder = new TextDecoder(); + let text = ""; + try { + while (!text.split("\n").includes(line)) { + const { done, value } = await reader.read(); + if (done) { + throw new Error(`fixture exited before printing ${line}`); + } + text += decoder.decode(value, { stream: true }); + } + } finally { + reader.releaseLock(); + } +} diff --git a/tests/e2e/shard.ts b/tests/e2e/shard.ts index 5d59d888a..fc800d2a7 100644 --- a/tests/e2e/shard.ts +++ b/tests/e2e/shard.ts @@ -65,6 +65,8 @@ export const E2E_DURATION_HINTS_SECONDS: Readonly> = { "native-config-files": 20, doctor: 18, "lifecycle-host-process": 13, + // Estimate until a hosted docker-e2e summary measures it: ~20 CLI ups/downs. + "shared-host-helpers": 40, "up-down": 12, // Estimate until a hosted docker-e2e summary measures it. "exec-stale-env": 20, diff --git a/tests/models/tla/README.md b/tests/models/tla/README.md index 816ebabca..dcc19f977 100644 --- a/tests/models/tla/README.md +++ b/tests/models/tla/README.md @@ -942,3 +942,110 @@ compatibility, resource improvements or release readiness. The model also omits implementation's graph-receipt/coordinator acknowledgement gap and startup release gate; those require their own implementation tests rather than an expanded claim about this unchanged specification. + +## Shared helper instance + +`shared-helper-instance/Instance.tla` models lifecycle-session ownership for one +shared-helper instance (one branch, HACK-1142): two overlapping `hack up` runs +(`a`, `b`) and one `hack down`, with the shared helper instance lock (`lock`), the +live session and its ownership token (`session`, `token`) and the persisted +lifecycle entry (`entry`, `healthy`) as explicit state. + +- `SessionSafe`: an up that succeeded with no down since it captured the shutdown + generation still has a live session, so an overlapping up's cleanup never kills a + session a successful up adopted. +- `Reclaimable`: a live session always has a persisted entry naming its token, so + the normal stop controller (`hack down`) can reclaim it. +- `NoForeignLaunch`: a start never launches a process into a session that carries + another start's token. +- `HelpersKeptForSession`: a down that could not confirm the session stopped keeps + the instance's helper holders while that session lives. + +| Model operation | Source boundary and regression | +| --- | --- | +| `Capture` / `Superseded` | `captureSharedHelperShutdownGeneration` before `up.before`; `startProjectSharedHelpers` refuses a changed generation before ensuring helpers. | +| `Decide` | The controller's session decision under the instance lock (`resolveSessionDisposition`): adopt an owned healthy session and record the adoption epoch, replace an owned stale (pending) one, refuse one without matching state, or decide to create. | +| `CreateBegin` | Start of the creation transaction (`ensureSession` → `createOwnedSession` under one `withLock`): `assertSessionStillAbsent` re-inspects and refuses unless the session is still absent. `tests/project-lifecycle-controller-ownership.test.ts` interleaving regression. | +| `Publish` / `CreateSession` | Pending entry published, then `createSession`; a failed creation removes only this start's pending entry (`removeLifecycleStateEntryIfOwned`). | +| `Record` / `RecordFail*` | tmux environment setup and `record("created")`, then the lock is released with the entry still pending. On failure the cleanup (`releaseFailedSessionCreation`: token read, name-based kill, entry removal) runs inside the same lock: the created session is stopped with its token before the entry is removed (`RecordFailStopped`); an unconfirmed (or timed-out) stop keeps the entry and token (`RecordFailUnconfirmed`). Controller regressions for both, including a hung `createSession`, a replacement overlapping the cleanup, and a replacement overlapping environment setup. | +| `StartProcesses` | Each launch (`startLifecycleProcess`) runs under the lock and only while the persisted entry and the live session still carry this start's token; otherwise the start is displaced (`LifecycleSessionDisplacedError`) and launches nothing. The pending entry is not adoptable, so another start may replace the session between launches. Controller regression: a displaced start's later process never reaches the replacement. | +| `Finalize` | `finalize()` under the lock (`publishFinalEntryIfStillOwned`): publish the final, adoptable entry only while the persisted entry and the live session still carry this start's created or adopted token; otherwise the start is displaced, gives up its authority and fails (`LifecycleSessionDisplacedError`). Controller regression: replacement after creation but before finalization, reclaimed by `stopLifecycleProcessController`. | +| `CreateTimeoutCreated` / `CreateTimeoutNotCreated` | A `createSession` that exceeds `muxCommandTimeoutMs()` is killed by `runMuxCommand`, so the call settles under the lock and throws `MuxCommandTimeoutError`: either the server created the session first (outcome unknown: entry and token kept for reclaim) or it did not (proven absent: entry retired). Controller regressions for both. | +| `InspectTimeout` | A timed-out listing, ownership-token read or window listing throws instead of returning an empty or absent result, and a listing that omits the session followed by an exact-presence query without a definite answer fails the inspection (`LifecycleSessionPresenceUnknownError`) instead of reading absence; the decision fails closed. Real tmux backend regressions with a hanging `tmux`, including that listing-omits-then-presence-times-out path; controller regression over a healthy owned session. | +| `CreateAbandon` / `LateCreate`, `ReplaceAbandon` / `LateKill`, `RecordFailReadUnlocked`, `MisreadDecide` / `MisreadCreateBegin` | Broken designs only (each enabled by one control): a deadline that stops waiting without cancelling the call, a failure cleanup whose kill lands after the lock was released, and a timeout read as absence. | +| `Launch` / `Succeed` / `Fail` | `assertCurrent` before and after containers start; Compose failure, SIGINT/SIGTERM or a failed check. | +| `Cleanup` / `CleanupUndecidable` | `SharedHelperSessionOwnership.runIfOwned`, which the combined operation cleanup and the controller's `abort()` / signal cleanup both go through: under the lock, a creator with a recorded epoch kills only while the epoch is its own; a creator whose record failed kills by token; the kill always checks the token. An unavailable lock keeps the session (and the helper holders: `LIFECYCLE_SESSION_RETAINED`). Controller regressions (abort overlapping a replacement; a signal callback already waiting on the lock while another up adopts; a startup whose second process fails with an unconfirmed stop, through `startLifecycleProcesses`), module tests and e2e overlap variants. | +| `DownMark` / `DownStop` / `DownStopFailed` | `markSharedHelperInstanceStopping` once the down is committed; the stop controller reclaims the session through the persisted entry; `releaseProjectSharedHelpers` advances the generation again. When the session stop failed or is unknown (including an inspection that could not tell whether the session exists), `stopSessionBeforeReleasingHelpers` keeps every holder, configured or recorded, and down, restart and native retirement fail (`DownStopFailed`). Unit regressions for both outcomes. | + +The positive configuration checks all four invariants and exhausts 7,649 distinct states +(TLC 1.7.4, Java 17, no fairness, `CHECK_DEADLOCK FALSE`). Each control disables one +fix through a constant and must fail with a named same-state witness: + +| Control | Fix disabled | Fails | Witness | +| --- | --- | --- | --- | +| `negative` | `CheckEpoch` (creator kills by token alone) | `SessionSafe` at `Succeed` | `a` created, `b` adopted, `a` failed and killed it, `b` succeeded: `session = "absent"`, `epoch = <<"b", "adopted">>` | +| `no-revalidation` | `Revalidate` | `Reclaimable` at `Publish` | `a` created and finalized meanwhile; `b`'s stale create decision overwrites the entry: `token = "a"`, `entry = "b"` | +| `epoch-failure-unstopped` | `StopOnEpochFailure` | `Reclaimable` at `RecordFailUnstopped` | the entry is removed while the created session lives: `token = "a"`, `entry = "none"` | +| `stale-finalize` | `FinalizeRecheck` | `Reclaimable` at `Finalize` | `b` replaced `a`'s pending session (token `b`); `a`'s stale finalize publishes its own token: `token = "b"`, `entry = "a"` | +| `abandoned-create` | `AbandonCreate` (call not settled before the lock is released) | `Reclaimable` at `LateCreate` | a down retires the pending entry while `a`'s creation is still running; it then creates the session: `token = "a"`, `entry = "none"` | +| `abandoned-replace` | `AbandonReplace` | `SessionSafe` at `LateKill` | `b`'s replacement released the lock with its stale read pending; `a` created and succeeded; `b`'s name-based kill stops `a`'s session | +| `timeout-as-absence` | `MisreadTimeout` | `Reclaimable` at `Publish` | a timed-out listing reads `a`'s live session as absent; `b` publishes over it: `token = "a"`, `entry = "b"` | +| `unlocked-failure-cleanup` | `UnlockedFailureCleanup` (cleanup kill after the lock is released) | `SessionSafe` at `LateKill` | `a`'s epoch record failed and its cleanup read the token; `b` replaced the pending session, created and succeeded; `a`'s name-based kill stops `b`'s session | +| `unfenced-launch` | `UnfencedLaunch` (launch by name without lock and token check) | `NoForeignLaunch` at `StartProcesses` | `b` replaced `a`'s pending session; `a`'s next launch lands in `b`'s session: `foreign = TRUE`, `token = "b"` | +| `release-on-failed-stop` | `ReleaseOnFailedStop` | `HelpersKeptForSession` at `DownStopFailed` | the down's session stop failed with `a`'s session alive; the holders are released anyway: `released = TRUE` | + +Bounds and assumptions: two ups, each running once; one down. `Decide`, the three +creation steps (with environment setup), `Record`, its failure cleanup, +`StartProcesses` and `Finalize` are lock-held steps; the container launch steps are +not. Environment setup has no step of its own because it runs inside the creation +transaction. `Decide` is one step because the source holds the lock +across inspect and record. Every mux call made by the lifecycle controller and the +tmux/zellij backends is bounded by `runMuxCommand`: the caller stops waiting +within the bound (a timed-out client is killed), and a timed-out call is unknown, +never absence. That does not prove the server will not still apply the command; +see the timeout residual below. The model covers tmux only: projects with shared +helpers refuse zellij for their lifecycle session, both before any hook runs +(`assertSharedHelperSessionMuxSupported`) and again at the backend the controller +actually selects (`MuxBackend.createsWithOwnerToken`), so a selection that changes +while hooks run is refused too, because zellij records the ownership +token with a separate `rename-tab` after creation, so an interrupted start could +leave a live session without a token that neither `hack up` nor `hack down` can +reclaim. tmux sets the token as part of `new-session` (`-e`), which is what +`CreateSession` and `CreateTimeoutCreated` assume. Helper ensure/holder counts, helper rollback +ownership (the operation record), the acquired-helper record and claim-level stop +confirmation are not modeled; the shared-host claim protocol has its own fixtures in +`tests/shared-host-lifecycle.test.ts`. `DownStop` joins the down's session stop and +its release into one step. In the implementation an up that captured after the +down's stopping mark can still succeed between the down's session stop and its +release; that is the documented first-slice limitation (launch prevention across a +concurrent down is detected after launch, not prevented). Real mux sessions, +process groups, lock-file durability and Compose behavior are outside this model. + +Timeout settlement is an assumption of the positive model, not a guarantee of the +code. `CreateTimeout*` treat a call as settled once `runMuxCommand` killed its +client. Killing the client does not prove that the tmux server will not still apply +the command: commands queue per client, so a command a stalled server already +received may run after it answered a later client. The `abandoned-create` and +`abandoned-replace` controls are that residual, and they fail. It needs a tmux +server that stalled past the deadline and then ran the dead client's command after +answering a later client; every call made while the server stays stalled also +times out and fails closed. What the code guarantees is weaker than the positive +model in that case. Consequences per late command, all for the same instance (one +project and branch): + +- create (`new-session`): a session that carries the failed start's token, possibly + without an entry. No lifecycle process runs in it, because launches only follow a + confirmed create. The next `hack up` refuses it ("already exists without matching + Hack lifecycle state"); stop it by hand. Helper holders are not affected. +- name-based kill (replacement, cleanup, down): it may stop a newer, successful + session of the instance, so `SessionSafe` does not hold for that start. Its entry + remains, so `hack down` and the next `hack up` reclaim or recreate it: recovery, + not prevention. +- environment, window or pane commands: they may reach a newer session of the + instance after its ownership check, so `NoForeignLaunch` does not hold for that + start; its `hack down` stops the whole session. + +Closing the residual needs a mutation fence rather than a timeout: target tmux +mutations by session ID (`#{session_id}`, never reused by a server) captured with +the token read. That fence, and full zellij support for shared-helper lifecycle +sessions, are tracked in HACK-1849 and are not in this slice. diff --git a/tests/models/tla/shared-helper-instance/Instance.tla b/tests/models/tla/shared-helper-instance/Instance.tla new file mode 100644 index 000000000..94901cd31 --- /dev/null +++ b/tests/models/tla/shared-helper-instance/Instance.tla @@ -0,0 +1,352 @@ +---- MODULE Instance ---- +\* Lifecycle-session ownership for one shared-helper instance (one branch): +\* two overlapping `hack up` runs and one `hack down`. Source mapping, bounds +\* and omissions are in tests/models/tla/README.md ("Shared helper instance"). +\* +\* Fix flags (all TRUE in positive.cfg; each control disables one): +\* CheckEpoch creator cleanup requires its own session epoch +\* Revalidate creation re-inspects the session under its lock +\* StopOnEpochFailure a failed epoch record stops the created session +\* before removing its entry +\* FinalizeRecheck finalize publishes only while the entry and the +\* live session still carry this start's token +\* +\* Broken-design flags (all FALSE in positive.cfg; each control sets one): +\* AbandonCreate a timed-out createSession releases the lock while the +\* call keeps running and may create the session later +\* AbandonReplace a timed-out replacement releases the lock while its +\* stale ownership read and name-based kill continue +\* MisreadTimeout a timed-out session listing reads as absence +\* UnlockedFailureCleanup a failed creation's cleanup reads the token +\* under the lock but kills by name after releasing it +\* UnfencedLaunch process launch (and environment setup) mutate the +\* session by name without the lock and token check +\* ReleaseOnFailedStop a down whose session stop failed or is unknown still +\* releases the instance's helper holders +EXTENDS Naturals, TLC +CONSTANTS CheckEpoch, Revalidate, StopOnEpochFailure, FinalizeRecheck, + AbandonCreate, AbandonReplace, MisreadTimeout, + UnlockedFailureCleanup, UnfencedLaunch, ReleaseOnFailedStop +Ups == {"a", "b"} +None == "none" +VARIABLES pc, cap, gen, epoch, owned, session, token, entry, healthy, lock, down, + mytok, stale, foreign, released +vars == <> + +Init == /\ pc = [u \in Ups |-> "idle"] + /\ cap = [u \in Ups |-> 0] + /\ gen = 0 + /\ epoch = <> + \* Cleanup authority: "no", "epoch" (created and recorded) or + \* "token" (created, epoch record failed, stop unconfirmed). + /\ owned = [u \in Ups |-> "no"] + /\ session = "absent" + /\ token = None \* the live session's ownership token + /\ entry = None \* persisted lifecycle entry's token + \* The entry is finalized with live processes (adoptable), not pending. + /\ healthy = FALSE + /\ lock = None \* shared helper instance lock holder + /\ down = "idle" + \* The session token this start created or adopted (finalize recheck). + /\ mytok = [u \in Ups |-> None] + \* Token an abandoned replacement read before its lock was released. + /\ stale = [u \in Ups |-> None] + \* A start launched a process into a session carrying another token. + /\ foreign = FALSE + \* The instance's helper holders were released (by a down) and not + \* acquired again since. + /\ released = FALSE + +Free == lock = None + +\* captureSharedHelperShutdownGeneration, before up.before hooks. +Capture(u) == /\ pc[u] = "idle" + /\ pc' = [pc EXCEPT ![u] = "captured"] + /\ cap' = [cap EXCEPT ![u] = gen] + /\ UNCHANGED <> + +\* startProjectSharedHelpers refuses a start a down superseded. +Superseded(u) == /\ pc[u] = "captured" /\ gen # cap[u] /\ Free + /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED <> + +\* Helpers ready, then the controller's session decision under the lock: +\* adopt an owned healthy session (recording the adoption epoch), refuse a +\* session without matching state, or decide to create. +Decide(u) == /\ pc[u] = "captured" /\ gen = cap[u] /\ Free + /\ IF session = "alive" /\ entry = token /\ healthy + THEN \* owned and healthy: adopt, recording the epoch + /\ pc' = [pc EXCEPT ![u] = "ready"] + /\ epoch' = <> + /\ mytok' = [mytok EXCEPT ![u] = token] + /\ UNCHANGED <> + ELSE IF session = "alive" /\ entry = token + THEN \* owned but stale (pending entry): replace it + /\ pc' = [pc EXCEPT ![u] = "create"] + /\ session' = "absent" /\ token' = None + /\ entry' = None /\ healthy' = FALSE + /\ UNCHANGED <> + ELSE IF session = "alive" + THEN \* no matching state: refuse (block) + /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED <> + ELSE /\ pc' = [pc EXCEPT ![u] = "create"] + /\ UNCHANGED <> + /\ released' = FALSE + /\ UNCHANGED <> + +\* Session creation transaction, one lock acquisition: revalidate, publish +\* pending ownership, create, record the epoch. +CreateBegin(u) == /\ pc[u] = "create" /\ Free + /\ IF Revalidate /\ session = "alive" + THEN /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED lock + ELSE /\ pc' = [pc EXCEPT ![u] = "publish"] + /\ lock' = u + /\ UNCHANGED <> + +Publish(u) == /\ pc[u] = "publish" /\ lock = u + /\ entry' = u /\ healthy' = FALSE + /\ pc' = [pc EXCEPT ![u] = "createsess"] + /\ UNCHANGED <> + +\* createSession fails when a session already exists; the pending entry is +\* removed only if it is still this start's (removeLifecycleStateEntryIfOwned). +CreateSession(u) == + /\ pc[u] = "createsess" /\ lock = u + /\ IF session = "alive" + THEN /\ entry' = IF entry = u THEN None ELSE entry + /\ lock' = None + /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED <> + ELSE /\ session' = "alive" /\ token' = u + /\ pc' = [pc EXCEPT ![u] = "record"] + /\ UNCHANGED <> + /\ UNCHANGED <> + +Record(u) == /\ pc[u] = "record" /\ lock = u + /\ epoch' = <> + /\ owned' = [owned EXCEPT ![u] = "epoch"] + /\ mytok' = [mytok EXCEPT ![u] = u] + /\ lock' = None + /\ pc' = [pc EXCEPT ![u] = "starting"] + /\ UNCHANGED <> + +\* Lifecycle processes start while the entry is still pending (not healthy), +\* so another start may replace the session between launches. Each launch +\* is fenced: under the lock, and only while the persisted entry and the +\* live session still carry this start's token; a displaced start gives up +\* its authority and fails. (tmux environment setup runs inside the locked +\* creation transaction, Publish..Record, so it needs no step of its own.) +\* The unfenced control launches by session name whatever it now holds. +StartProcesses(u) == + /\ pc[u] = "starting" + /\ IF UnfencedLaunch + THEN /\ pc' = [pc EXCEPT ![u] = "finalize"] + /\ foreign' = (foreign \/ (session = "alive" /\ token # mytok[u])) + /\ UNCHANGED owned + ELSE /\ Free + /\ IF entry = mytok[u] /\ token = mytok[u] + THEN /\ pc' = [pc EXCEPT ![u] = "finalize"] + /\ UNCHANGED owned + ELSE /\ owned' = [owned EXCEPT ![u] = "no"] + /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED foreign + /\ UNCHANGED <> + +\* finalize() under the lock: publish the final (healthy) entry only while the +\* persisted entry and the live session still carry this start's token; +\* otherwise the start was displaced, gives up its authority and fails. +Finalize(u) == + /\ pc[u] = "finalize" /\ Free + /\ IF ~FinalizeRecheck \/ (entry = mytok[u] /\ token = mytok[u]) + THEN /\ entry' = mytok[u] /\ healthy' = TRUE + /\ pc' = [pc EXCEPT ![u] = "ready"] + /\ UNCHANGED owned + ELSE /\ owned' = [owned EXCEPT ![u] = "no"] + /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED <> + /\ UNCHANGED <> + +\* The epoch record fails after the session was created. Its cleanup (token +\* read, name-based kill, entry removal) runs inside the creation lock. +RecordFailStopped(u) == + /\ StopOnEpochFailure /\ ~UnlockedFailureCleanup + /\ pc[u] = "record" /\ lock = u + /\ session' = "absent" /\ token' = None + /\ entry' = IF entry = u THEN None ELSE entry + /\ lock' = None /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED <> +\* Control: the cleanup reads the token under the lock, releases it, and the +\* name-based kill lands later (LateKill), possibly on a replacement. +RecordFailReadUnlocked(u) == + /\ StopOnEpochFailure /\ UnlockedFailureCleanup + /\ pc[u] = "record" /\ lock = u + /\ stale' = [stale EXCEPT ![u] = token] + /\ lock' = None /\ pc' = [pc EXCEPT ![u] = "pendingkill"] + /\ UNCHANGED <> +RecordFailUnconfirmed(u) == + /\ StopOnEpochFailure /\ pc[u] = "record" /\ lock = u + /\ owned' = [owned EXCEPT ![u] = "token"] + /\ lock' = None /\ pc' = [pc EXCEPT ![u] = "failed"] + /\ UNCHANGED <> +RecordFailUnstopped(u) == + /\ ~StopOnEpochFailure /\ pc[u] = "record" /\ lock = u + /\ entry' = IF entry = u THEN None ELSE entry + /\ lock' = None /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED <> + +\* assertCurrent before containers start. +Launch(u) == /\ pc[u] = "ready" /\ gen = cap[u] /\ Free + /\ pc' = [pc EXCEPT ![u] = "launched"] + /\ UNCHANGED <> + +\* assertCurrent after containers start. +Succeed(u) == /\ pc[u] = "launched" /\ gen = cap[u] /\ Free + /\ pc' = [pc EXCEPT ![u] = "succeeded"] + /\ UNCHANGED <> + +\* Compose failure, SIGINT/SIGTERM, or a failed assertCurrent. +Fail(u) == /\ pc[u] \in {"starting", "finalize", "ready", "launched"} + /\ pc' = [pc EXCEPT ![u] = "failed"] + /\ UNCHANGED <> + +\* runIfOwned under the lock, then the token-checked kill and entry removal. +Authorized(u) == \/ owned[u] = "token" + \/ /\ owned[u] = "epoch" + /\ (~CheckEpoch \/ epoch = <>) +Cleanup(u) == /\ pc[u] = "failed" /\ Free + /\ pc' = [pc EXCEPT ![u] = "done"] + /\ IF Authorized(u) /\ session = "alive" /\ token = u + THEN /\ session' = "absent" /\ token' = None + /\ entry' = IF entry = u THEN None ELSE entry + ELSE UNCHANGED <> + /\ UNCHANGED <> + +\* The instance lock stayed busy: ownership undecidable, session kept. +CleanupUndecidable(u) == /\ pc[u] = "failed" + /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED <> + +\* A committed down: the stopping mark advances the generation; then the +\* normal stop controller stops the session through the persisted entry +\* (it can only reclaim a session whose entry names its token) and the +\* release advances the generation again (one step here; see README). +DownMark == /\ down = "idle" /\ Free + /\ down' = "marked" /\ gen' = gen + 1 + /\ UNCHANGED <> +DownStop == /\ down = "marked" /\ Free + /\ down' = "stopped" /\ gen' = gen + 1 + /\ IF session = "alive" /\ entry = token + THEN /\ session' = "absent" /\ token' = None + ELSE UNCHANGED <> + /\ entry' = None /\ healthy' = FALSE /\ released' = TRUE + /\ UNCHANGED <> +\* The session stop failed or its outcome is unknown, including an inspection +\* that could not tell whether the session exists (a timed-out listing or +\* exact-presence query): the session and its entry stay, and so do the +\* helper holders (the down fails; a later down releases both). It can happen +\* whether or not the session is actually alive. The control releases the +\* holders anyway. +DownStopFailed == /\ down = "marked" /\ Free + /\ down' = "failed" + /\ IF ReleaseOnFailedStop + THEN /\ gen' = gen + 1 /\ released' = TRUE + ELSE UNCHANGED <> + /\ UNCHANGED <> + +\* Bounded mux calls settle under the lock (positive design): a timed-out +\* createSession was killed; the server either created the session first +\* (outcome unknown: keep the entry and token authority) or not (retire it). +CreateTimeoutCreated(u) == + /\ ~AbandonCreate /\ pc[u] = "createsess" /\ lock = u /\ session = "absent" + /\ session' = "alive" /\ token' = u + /\ owned' = [owned EXCEPT ![u] = "token"] + /\ lock' = None /\ pc' = [pc EXCEPT ![u] = "failed"] + /\ UNCHANGED <> +CreateTimeoutNotCreated(u) == + /\ ~AbandonCreate /\ pc[u] = "createsess" /\ lock = u /\ session = "absent" + /\ entry' = IF entry = u THEN None ELSE entry + /\ lock' = None /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED <> + +\* Broken design: the deadline stops waiting but the call keeps running. +CreateAbandon(u) == + /\ AbandonCreate /\ pc[u] = "createsess" /\ lock = u /\ session = "absent" + /\ lock' = None /\ pc' = [pc EXCEPT ![u] = "pending"] + /\ UNCHANGED <> +LateCreate(u) == + /\ pc[u] = "pending" + /\ IF session = "absent" + THEN /\ session' = "alive" /\ token' = u + ELSE UNCHANGED <> + /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED <> + +\* Broken design: a stale (pending) session's replacement reads its token, +\* then its deadline releases the lock; the kill and retirement continue. +ReplaceAbandon(u) == + /\ AbandonReplace /\ pc[u] = "captured" /\ gen = cap[u] /\ Free + /\ session = "alive" /\ entry = token /\ ~healthy + /\ stale' = [stale EXCEPT ![u] = token] + /\ pc' = [pc EXCEPT ![u] = "pendingkill"] + /\ UNCHANGED <> +LateKill(u) == + /\ pc[u] = "pendingkill" + \* The stale read matched, so the kill goes out by session name. + /\ session' = "absent" /\ token' = None + /\ entry' = IF entry = stale[u] THEN None ELSE entry + /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED <> + +\* A timed-out inspection (positive design): unknown, fail closed. Covers a +\* timed-out listing and a listing that omits the session followed by an +\* exact-presence query without a definite answer +\* (LifecycleSessionPresenceUnknownError); the up refuses. +InspectTimeout(u) == + /\ pc[u] \in {"captured", "create"} /\ Free + /\ pc' = [pc EXCEPT ![u] = "done"] + /\ UNCHANGED <> +\* Broken design: a timed-out listing reads as absence, both in the decision +\* and in the creation transaction's revalidation. +MisreadDecide(u) == + /\ MisreadTimeout /\ pc[u] = "captured" /\ gen = cap[u] /\ Free + /\ session = "alive" + /\ pc' = [pc EXCEPT ![u] = "create"] + /\ UNCHANGED <> +MisreadCreateBegin(u) == + /\ MisreadTimeout /\ pc[u] = "create" /\ Free /\ session = "alive" + /\ pc' = [pc EXCEPT ![u] = "publish"] /\ lock' = u + /\ UNCHANGED <> + +Next == \/ \E u \in Ups : + \/ Capture(u) \/ Superseded(u) \/ Decide(u) + \/ CreateBegin(u) \/ Publish(u) \/ CreateSession(u) + \/ Record(u) \/ StartProcesses(u) \/ Finalize(u) + \/ RecordFailStopped(u) \/ RecordFailReadUnlocked(u) + \/ RecordFailUnconfirmed(u) \/ RecordFailUnstopped(u) + \/ CreateTimeoutCreated(u) \/ CreateTimeoutNotCreated(u) + \/ CreateAbandon(u) \/ LateCreate(u) + \/ ReplaceAbandon(u) \/ LateKill(u) + \/ InspectTimeout(u) \/ MisreadDecide(u) \/ MisreadCreateBegin(u) + \/ Launch(u) \/ Succeed(u) \/ Fail(u) + \/ Cleanup(u) \/ CleanupUndecidable(u) + \/ DownMark \/ DownStop \/ DownStopFailed +Spec == Init /\ [][Next]_vars + +\* An up that succeeded with no down since its capture keeps a live session. +SessionSafe == \A u \in Ups : + (pc[u] = "succeeded" /\ cap[u] = gen) => session = "alive" + +\* A live session always has a persisted entry naming its token, so the +\* normal stop controller (hack down) can reclaim it. +Reclaimable == session = "alive" => entry = token + +\* A start never mutates (launches into) a session carrying another token. +NoForeignLaunch == ~foreign + +\* A down that could not confirm the session stopped keeps the helper +\* holders that session's processes may still use. +HelpersKeptForSession == (down = "failed" /\ session = "alive") => ~released +==== diff --git a/tests/models/tla/shared-helper-instance/abandoned-create.cfg b/tests/models/tla/shared-helper-instance/abandoned-create.cfg new file mode 100644 index 000000000..751bcc40c --- /dev/null +++ b/tests/models/tla/shared-helper-instance/abandoned-create.cfg @@ -0,0 +1,14 @@ +SPECIFICATION Spec +CONSTANTS + CheckEpoch = TRUE + Revalidate = TRUE + StopOnEpochFailure = TRUE + FinalizeRecheck = TRUE + AbandonCreate = TRUE + AbandonReplace = FALSE + MisreadTimeout = FALSE + UnlockedFailureCleanup = FALSE + UnfencedLaunch = FALSE + ReleaseOnFailedStop = FALSE +INVARIANT Reclaimable +CHECK_DEADLOCK FALSE diff --git a/tests/models/tla/shared-helper-instance/abandoned-replace.cfg b/tests/models/tla/shared-helper-instance/abandoned-replace.cfg new file mode 100644 index 000000000..16d0600ae --- /dev/null +++ b/tests/models/tla/shared-helper-instance/abandoned-replace.cfg @@ -0,0 +1,14 @@ +SPECIFICATION Spec +CONSTANTS + CheckEpoch = TRUE + Revalidate = TRUE + StopOnEpochFailure = TRUE + FinalizeRecheck = TRUE + AbandonCreate = FALSE + AbandonReplace = TRUE + MisreadTimeout = FALSE + UnlockedFailureCleanup = FALSE + UnfencedLaunch = FALSE + ReleaseOnFailedStop = FALSE +INVARIANT SessionSafe +CHECK_DEADLOCK FALSE diff --git a/tests/models/tla/shared-helper-instance/epoch-failure-unstopped.cfg b/tests/models/tla/shared-helper-instance/epoch-failure-unstopped.cfg new file mode 100644 index 000000000..41fcc167c --- /dev/null +++ b/tests/models/tla/shared-helper-instance/epoch-failure-unstopped.cfg @@ -0,0 +1,14 @@ +SPECIFICATION Spec +CONSTANTS + CheckEpoch = TRUE + Revalidate = TRUE + StopOnEpochFailure = FALSE + FinalizeRecheck = TRUE + AbandonCreate = FALSE + AbandonReplace = FALSE + MisreadTimeout = FALSE + UnlockedFailureCleanup = FALSE + UnfencedLaunch = FALSE + ReleaseOnFailedStop = FALSE +INVARIANT Reclaimable +CHECK_DEADLOCK FALSE diff --git a/tests/models/tla/shared-helper-instance/negative.cfg b/tests/models/tla/shared-helper-instance/negative.cfg new file mode 100644 index 000000000..06cf1114a --- /dev/null +++ b/tests/models/tla/shared-helper-instance/negative.cfg @@ -0,0 +1,14 @@ +SPECIFICATION Spec +CONSTANTS + CheckEpoch = FALSE + Revalidate = TRUE + StopOnEpochFailure = TRUE + FinalizeRecheck = TRUE + AbandonCreate = FALSE + AbandonReplace = FALSE + MisreadTimeout = FALSE + UnlockedFailureCleanup = FALSE + UnfencedLaunch = FALSE + ReleaseOnFailedStop = FALSE +INVARIANT SessionSafe +CHECK_DEADLOCK FALSE diff --git a/tests/models/tla/shared-helper-instance/no-revalidation.cfg b/tests/models/tla/shared-helper-instance/no-revalidation.cfg new file mode 100644 index 000000000..ce41a6946 --- /dev/null +++ b/tests/models/tla/shared-helper-instance/no-revalidation.cfg @@ -0,0 +1,14 @@ +SPECIFICATION Spec +CONSTANTS + CheckEpoch = TRUE + Revalidate = FALSE + StopOnEpochFailure = TRUE + FinalizeRecheck = TRUE + AbandonCreate = FALSE + AbandonReplace = FALSE + MisreadTimeout = FALSE + UnlockedFailureCleanup = FALSE + UnfencedLaunch = FALSE + ReleaseOnFailedStop = FALSE +INVARIANT Reclaimable +CHECK_DEADLOCK FALSE diff --git a/tests/models/tla/shared-helper-instance/positive.cfg b/tests/models/tla/shared-helper-instance/positive.cfg new file mode 100644 index 000000000..42a2e5f20 --- /dev/null +++ b/tests/models/tla/shared-helper-instance/positive.cfg @@ -0,0 +1,14 @@ +SPECIFICATION Spec +CONSTANTS + CheckEpoch = TRUE + Revalidate = TRUE + StopOnEpochFailure = TRUE + FinalizeRecheck = TRUE + AbandonCreate = FALSE + AbandonReplace = FALSE + MisreadTimeout = FALSE + UnlockedFailureCleanup = FALSE + UnfencedLaunch = FALSE + ReleaseOnFailedStop = FALSE +INVARIANTS SessionSafe Reclaimable NoForeignLaunch HelpersKeptForSession +CHECK_DEADLOCK FALSE diff --git a/tests/models/tla/shared-helper-instance/release-on-failed-stop.cfg b/tests/models/tla/shared-helper-instance/release-on-failed-stop.cfg new file mode 100644 index 000000000..1bfc7cc74 --- /dev/null +++ b/tests/models/tla/shared-helper-instance/release-on-failed-stop.cfg @@ -0,0 +1,14 @@ +SPECIFICATION Spec +CONSTANTS + CheckEpoch = TRUE + Revalidate = TRUE + StopOnEpochFailure = TRUE + FinalizeRecheck = TRUE + AbandonCreate = FALSE + AbandonReplace = FALSE + MisreadTimeout = FALSE + UnlockedFailureCleanup = FALSE + UnfencedLaunch = FALSE + ReleaseOnFailedStop = TRUE +INVARIANT HelpersKeptForSession +CHECK_DEADLOCK FALSE diff --git a/tests/models/tla/shared-helper-instance/stale-finalize.cfg b/tests/models/tla/shared-helper-instance/stale-finalize.cfg new file mode 100644 index 000000000..52fddf782 --- /dev/null +++ b/tests/models/tla/shared-helper-instance/stale-finalize.cfg @@ -0,0 +1,14 @@ +SPECIFICATION Spec +CONSTANTS + CheckEpoch = TRUE + Revalidate = TRUE + StopOnEpochFailure = TRUE + FinalizeRecheck = FALSE + AbandonCreate = FALSE + AbandonReplace = FALSE + MisreadTimeout = FALSE + UnlockedFailureCleanup = FALSE + UnfencedLaunch = FALSE + ReleaseOnFailedStop = FALSE +INVARIANT Reclaimable +CHECK_DEADLOCK FALSE diff --git a/tests/models/tla/shared-helper-instance/timeout-as-absence.cfg b/tests/models/tla/shared-helper-instance/timeout-as-absence.cfg new file mode 100644 index 000000000..bd5ef1a85 --- /dev/null +++ b/tests/models/tla/shared-helper-instance/timeout-as-absence.cfg @@ -0,0 +1,14 @@ +SPECIFICATION Spec +CONSTANTS + CheckEpoch = TRUE + Revalidate = TRUE + StopOnEpochFailure = TRUE + FinalizeRecheck = TRUE + AbandonCreate = FALSE + AbandonReplace = FALSE + MisreadTimeout = TRUE + UnlockedFailureCleanup = FALSE + UnfencedLaunch = FALSE + ReleaseOnFailedStop = FALSE +INVARIANT Reclaimable +CHECK_DEADLOCK FALSE diff --git a/tests/models/tla/shared-helper-instance/unfenced-launch.cfg b/tests/models/tla/shared-helper-instance/unfenced-launch.cfg new file mode 100644 index 000000000..65335360c --- /dev/null +++ b/tests/models/tla/shared-helper-instance/unfenced-launch.cfg @@ -0,0 +1,14 @@ +SPECIFICATION Spec +CONSTANTS + CheckEpoch = TRUE + Revalidate = TRUE + StopOnEpochFailure = TRUE + FinalizeRecheck = TRUE + AbandonCreate = FALSE + AbandonReplace = FALSE + MisreadTimeout = FALSE + UnlockedFailureCleanup = FALSE + UnfencedLaunch = TRUE + ReleaseOnFailedStop = FALSE +INVARIANT NoForeignLaunch +CHECK_DEADLOCK FALSE diff --git a/tests/models/tla/shared-helper-instance/unlocked-failure-cleanup.cfg b/tests/models/tla/shared-helper-instance/unlocked-failure-cleanup.cfg new file mode 100644 index 000000000..5a2cc3016 --- /dev/null +++ b/tests/models/tla/shared-helper-instance/unlocked-failure-cleanup.cfg @@ -0,0 +1,14 @@ +SPECIFICATION Spec +CONSTANTS + CheckEpoch = TRUE + Revalidate = TRUE + StopOnEpochFailure = TRUE + FinalizeRecheck = TRUE + AbandonCreate = FALSE + AbandonReplace = FALSE + MisreadTimeout = FALSE + UnlockedFailureCleanup = TRUE + UnfencedLaunch = FALSE + ReleaseOnFailedStop = FALSE +INVARIANT SessionSafe +CHECK_DEADLOCK FALSE diff --git a/tests/mux-command-timeout.test.ts b/tests/mux-command-timeout.test.ts new file mode 100644 index 000000000..fd48a5829 --- /dev/null +++ b/tests/mux-command-timeout.test.ts @@ -0,0 +1,141 @@ +import { afterEach, expect, test } from "bun:test"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { stopSessionBeforeReleasingHelpers } from "../src/commands/project.ts"; +import { + readLifecycleState, + upsertLifecycleStateEntry, +} from "../src/lib/lifecycle-runtime.ts"; +import { stopLifecycleProcessController } from "../src/lib/project-lifecycle-controller.ts"; +import { + inspectLifecycleSession, + LifecycleSessionPresenceUnknownError, +} from "../src/lib/project-lifecycle-sessions.ts"; +import { MuxCommandTimeoutError } from "../src/mux/mux-backend.ts"; +import { createTmuxBackend } from "../src/mux/tmux-backend.ts"; + +/** + * A timed-out tmux command is killed and reported as an unknown outcome, + * never as an empty or absent result: a hung `tmux list-sessions` must not + * read as "no sessions", or up/down would overwrite or retire a healthy + * owned session's state. + */ + +const roots: string[] = []; +const originalPath = process.env.PATH; +const originalTimeout = process.env.HACK_MUX_COMMAND_TIMEOUT_MS; + +afterEach(async () => { + process.env.PATH = originalPath; + if (originalTimeout === undefined) { + Reflect.deleteProperty(process.env, "HACK_MUX_COMMAND_TIMEOUT_MS"); + } else { + process.env.HACK_MUX_COMMAND_TIMEOUT_MS = originalTimeout; + } + for (const root of roots.splice(0)) { + await rm(root, { recursive: true, force: true }); + } +}); + +async function hangingTmux(): Promise { + const root = await mkdtemp(join(tmpdir(), "hack-mux-timeout-")); + roots.push(root); + const bin = join(root, "bin"); + await mkdir(bin, { recursive: true }); + // Every query hangs, like a wedged tmux server. + await writeFile(join(bin, "tmux"), "#!/bin/sh\nsleep 30\n", { + mode: 0o755, + }); + process.env.PATH = `${bin}:${originalPath ?? ""}`; + process.env.HACK_MUX_COMMAND_TIMEOUT_MS = "200"; +} + +test("a timed-out tmux session listing throws instead of returning no sessions", async () => { + await hangingTmux(); + const backend = createTmuxBackend(); + const started = performance.now(); + await expect(backend.listSessions()).rejects.toBeInstanceOf( + MuxCommandTimeoutError + ); + expect(performance.now() - started).toBeLessThan(5000); +}); + +test("a timed-out tmux presence query is unknown, and an ownership-token read throws", async () => { + await hangingTmux(); + const backend = createTmuxBackend(); + expect(await backend.readSessionPresence?.({ name: "demo" })).toBe("unknown"); + await expect( + backend.readLifecycleOwnerToken?.({ name: "demo" }) + ).rejects.toBeInstanceOf(MuxCommandTimeoutError); +}); + +test("a listing that omits the session plus a timed-out presence query is unknown: down keeps the entry and helpers", async () => { + const root = await mkdtemp(join(tmpdir(), "hack-mux-unlisted-")); + roots.push(root); + const bin = join(root, "bin"); + await mkdir(bin, { recursive: true }); + // The listing answers without the session; the exact query hangs. + await writeFile( + join(bin, "tmux"), + '#!/bin/sh\ncase "$*" in *list-sessions*) exit 0 ;; *has-session*) sleep 30 ;; esac\nexit 1\n', + { mode: 0o755 } + ); + process.env.PATH = `${bin}:${originalPath ?? ""}`; + process.env.HACK_MUX_COMMAND_TIMEOUT_MS = "200"; + const backend = createTmuxBackend(); + const projectDir = join(root, ".hack"); + await mkdir(projectDir, { recursive: true }); + const entry = { + composeProject: "demo", + projectName: "demo", + branch: null, + sessionName: "demo--lifecycle", + backend: "tmux" as const, + ownershipToken: "token-a", + definitionHash: "sha256:test", + processes: [], + updatedAt: new Date().toISOString(), + }; + await upsertLifecycleStateEntry({ projectDir, entry }); + // The inspection boundary: unknown, never absence (which would decide + // "create" and let creation revalidation pass). + await expect( + inspectLifecycleSession({ + backend, + entry, + expectedSessionName: entry.sessionName, + expectedProjectRoot: root, + expectedDefinitionHash: entry.definitionHash, + }) + ).rejects.toBeInstanceOf(LifecycleSessionPresenceUnknownError); + // hack down's stop: fails, keeps the entry, and the helpers stay held. + const outcome = await stopSessionBeforeReleasingHelpers({ + stopSession: () => + stopLifecycleProcessController({ + project: { projectRoot: root, projectDir }, + definitionHash: entry.definitionHash, + projectName: "demo", + branch: null, + composeProject: entry.composeProject, + mux: () => + Promise.resolve({ + mode: "tmux", + backends: new Map([["tmux", backend]]), + }), + }), + containers: "stopped", + releaseHelpers: () => Promise.resolve([]), + heldHelperNames: () => Promise.resolve(["mail"]), + }); + expect(outcome.sessionStopFailed?.error).toBeInstanceOf( + LifecycleSessionPresenceUnknownError + ); + expect(outcome.notReleased).toEqual(["mail"]); + expect( + (await readLifecycleState({ projectDir })).map( + (kept) => kept.ownershipToken + ) + ).toEqual(["token-a"]); +}); diff --git a/tests/native-project-down.test.ts b/tests/native-project-down.test.ts index df41b036f..79c1e8648 100644 --- a/tests/native-project-down.test.ts +++ b/tests/native-project-down.test.ts @@ -1,6 +1,13 @@ import { afterEach, expect, test } from "bun:test"; import { createHash } from "node:crypto"; -import { mkdir, mkdtemp, realpath, rm, writeFile } from "node:fs/promises"; +import { + mkdir, + mkdtemp, + readFile, + realpath, + rm, + writeFile, +} from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { @@ -17,6 +24,10 @@ import { removeNativeProjectRun, saveNativeProjectRun, } from "../src/backends/native-project-run.ts"; +import { + markSharedHelperInstanceStopping, + type ProjectSharedHelperConfig, +} from "../src/lib/project-shared-helpers.ts"; import { expectClosedRefusal } from "./helpers/closed-refusal.ts"; import { privateCliEnv } from "./helpers/private-cli-env.ts"; @@ -864,3 +875,123 @@ test("host retirement failure preserves confirmed stopped state and skips after ).rejects.toThrow("host ownership uncertain"); expect(await loadNativeProjectRun(opts.scope)).toEqual(run); }); + +/** + * Native down marks shared helpers as stopping only once the down is + * committed: after its before hook and the ownership/finalization rechecks, + * immediately before graph cleanup. A down refused at any of those boundaries + * must leave the shared helper shutdown generation and operation record alone, + * so a concurrent up is not superseded by a shutdown that never happened. + */ +async function sharedHelperInstance(projectRoot: string) { + const hackDir = join(projectRoot, "hack-home"); + const instances = join(hackDir, "shared-helpers", ".instances"); + await mkdir(instances, { recursive: true, mode: 0o700 }); + const operationPath = join(instances, "demo--a.operation"); + await writeFile(operationPath, "current-up-operation\n", { mode: 0o600 }); + const helper: ProjectSharedHelperConfig = { + name: "issuer-bridge", + enabled: true, + command: ["/bin/true"], + port: 4100, + identityPath: "/__hack/identity", + upstream: "https://issuer.test", + env: [], + readyTimeoutMs: 1000, + }; + return { + mark: async () => { + await markSharedHelperInstanceStopping({ + helpers: [helper], + composeProject: "demo--a", + hackDir, + }); + }, + generation: () => + readFile(join(instances, "demo--a.shutdown-generation"), "utf8").catch( + () => null + ), + operation: () => readFile(operationPath, "utf8"), + }; +} + +test("a native down refused after its before hook leaves the shared helper shutdown state unchanged", async () => { + for (const refusal of ["finalization", "ownership", "hook"] as const) { + const opts = await fixture(); + const helpers = await sharedHelperInstance(opts.scope.projectRoot); + const calls: string[] = []; + let inspections = 0; + const result = nativeProjectDown({ + ...opts, + before: async () => { + calls.push("before"); + if (refusal === "finalization") { + await beginNativeProjectFinalization({ scope: opts.scope, run }); + } + if (refusal === "hook") { + throw new Error("down.before hook failed"); + } + }, + beforeCleanup: async () => { + calls.push("before-cleanup"); + await helpers.mark(); + }, + invoke: async (request) => { + calls.push(request.args[1]!); + inspections += 1; + const value = snapshot(); + if (refusal === "ownership" && inspections > 1) { + value.receipt.owner = "f".repeat(32); + } + return value; + }, + }); + if (refusal === "hook") { + await expect(result).rejects.toThrow("down.before hook failed"); + } else { + await expectClosedRefusal( + result, + refusal === "finalization" ? "ownership changed" : "unconfirmed" + ); + } + expect(calls).not.toContain("before-cleanup"); + expect(calls).not.toContain("cleanup"); + expect(await helpers.generation()).toBeNull(); + expect(await helpers.operation()).toBe("current-up-operation\n"); + } +}); + +test("a committed native down marks shared helpers as stopping immediately before graph cleanup", async () => { + const opts = await fixture(); + const helpers = await sharedHelperInstance(opts.scope.projectRoot); + const calls: string[] = []; + let cleaned = false; + const result = await nativeProjectDown({ + ...opts, + before: async () => { + calls.push("before"); + }, + beforeCleanup: async () => { + calls.push("before-cleanup"); + await helpers.mark(); + }, + invoke: async (request) => { + calls.push(request.args[1]!); + if (request.args[1] === "cleanup") { + cleaned = true; + return {}; + } + return snapshot(cleaned); + }, + }); + expect(result.status).toBe("stopped"); + expect(calls.slice(0, 5)).toEqual([ + "inspect", + "before", + "inspect", + "before-cleanup", + "cleanup", + ]); + expect(await helpers.generation()).not.toBeNull(); + expect(await helpers.operation()).toStartWith("stopping:"); +}); diff --git a/tests/project-config-schema.test.ts b/tests/project-config-schema.test.ts index 2e0a6f837..9999b0389 100644 --- a/tests/project-config-schema.test.ts +++ b/tests/project-config-schema.test.ts @@ -155,3 +155,31 @@ test("managed env schema models entry kind, metadata, and service scope", () => }, ]); }); + +test("project config schema declares lifecycle.sharedHelpers", () => { + const schema = JSON.parse(renderProjectConfigSchemaJson()) as { + properties: { + lifecycle: { properties: Record> }; + }; + }; + const sharedHelpers = schema.properties.lifecycle.properties.sharedHelpers; + const items = sharedHelpers?.items as Record; + expect(items.additionalProperties).toBe(false); + expect(items.required).toEqual(["name", "command", "port"]); + expect(items.oneOf).toEqual([ + { required: ["upstream"] }, + { required: ["upstreamEnv"] }, + ]); + expect(Object.keys(items.properties as object).sort()).toEqual([ + "command", + "cwd", + "enabled", + "env", + "identityPath", + "name", + "port", + "readyTimeoutMs", + "upstream", + "upstreamEnv", + ]); +}); diff --git a/tests/project-lifecycle-controller-ownership.test.ts b/tests/project-lifecycle-controller-ownership.test.ts new file mode 100644 index 000000000..375d7885a --- /dev/null +++ b/tests/project-lifecycle-controller-ownership.test.ts @@ -0,0 +1,886 @@ +import { afterEach, expect, test } from "bun:test"; +import { appendFileSync } from "node:fs"; +import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + assertSharedHelperSessionMuxSupported, + releaseAfterFailedProcessStart, + startLifecycleProcesses, +} from "../src/commands/project.ts"; +import { + readLifecycleState, + upsertLifecycleStateEntry, +} from "../src/lib/lifecycle-runtime.ts"; +import { + createLifecycleProcessController, + LifecycleSessionDisplacedError, + LifecycleSessionRetainedError, + stopLifecycleProcessController, +} from "../src/lib/project-lifecycle-controller.ts"; +import { createSharedHelperSessionOwnership } from "../src/lib/project-shared-helpers.ts"; +import { + type MuxBackend, + MuxCommandTimeoutError, + type MuxSession, + type MuxSessionCreateResult, +} from "../src/mux/mux-backend.ts"; +import { setLoggerBackendOverride } from "../src/ui/logger.ts"; + +/** + * Lifecycle-session creation for projects with shared helpers, driven + * against the real controller with an in-memory mux backend. Its sessions + * carry their ownership token from creation, as tmux's do (`new-session -e`); + * it borrows the zellij name only for the simple `zellij run` launch path. + * Real zellij writes the token with a separate `rename-tab`, so projects with + * shared helpers refuse it (assertSharedHelperSessionMuxSupported). + * - a failure after the session was created (including a failed session + * epoch record) never leaves a session without an ownership record; + * - a start that decided to create, then lost to an overlapping start that + * created and finalized the session, refuses without touching the + * winner's entry, which stays reclaimable. + */ + +const roots: string[] = []; + +afterEach(async () => { + setLoggerBackendOverride({ backend: null }); + for (const root of roots.splice(0)) { + await rm(root, { recursive: true, force: true }); + } +}); + +const COMPOSE_PROJECT = "demo--branch-a"; +const SESSION = "demo--lifecycle-branch-a"; + +type FakeMux = Omit & { + readonly sessions: Map; + failKill: boolean; + /** Mutable so a test can switch the selected backend's capability. */ + createsWithOwnerToken: boolean; + /** + * createSession times out (the backend kills the client and throws) + * after the session was created server-side, or without creating it. + */ + timeoutCreate: "created" | "not-created" | null; + /** readLifecycleOwnerToken times out. */ + timeoutToken: boolean; + /** listSessions times out. */ + timeoutList: boolean; + /** This token read (1-based) stalls, then returns the value read at call time. */ + stallTokenRead: number; + tokenReads: number; + readonly killed: string[]; + /** Mux calls still running: a bounded call must settle before return. */ + inFlight: number; + /** When set, session creations and kills are appended here (one per line). */ + eventsPath: string | null; +}; + +const TIMEOUT_DELAY_MS = 50; +/** Long enough that an overlapping start could finish meanwhile. */ +const STALLED_READ_MS = 1500; + +function fakeMux(backendName: "zellij" | "tmux" = "zellij"): FakeMux { + const sessions = new Map(); + const session = (name: string): MuxSession => ({ + backend: backendName, + name, + attached: false, + path: null, + windows: 1, + createdAt: null, + }); + const track = async (work: () => Promise): Promise => { + fake.inFlight += 1; + try { + return await work(); + } finally { + fake.inFlight -= 1; + } + }; + const fake: FakeMux = { + name: backendName, + available: true, + // Its sessions carry their token from creation, as tmux's do. + createsWithOwnerToken: true, + sessions, + failKill: false, + timeoutCreate: null, + timeoutToken: false, + timeoutList: false, + stallTokenRead: 0, + tokenReads: 0, + killed: [], + inFlight: 0, + eventsPath: null, + listSessions: () => + track(async () => { + if (fake.timeoutList) { + await Bun.sleep(TIMEOUT_DELAY_MS); + throw new MuxCommandTimeoutError(["zellij", "list-sessions"]); + } + return [...sessions.keys()].map((name) => session(name)); + }), + createSession: (opts): Promise => + track(async () => { + if (fake.timeoutCreate) { + if (fake.timeoutCreate === "created") { + sessions.set(opts.name, opts.lifecycleOwnerToken ?? ""); + } + await Bun.sleep(TIMEOUT_DELAY_MS); + throw new MuxCommandTimeoutError(["zellij", "attach"]); + } + if (sessions.has(opts.name)) { + return { ok: false, error: "session exists" }; + } + sessions.set(opts.name, opts.lifecycleOwnerToken ?? ""); + if (fake.eventsPath) { + appendFileSync( + fake.eventsPath, + `create ${opts.lifecycleOwnerToken}\n` + ); + } + return { ok: true, session: session(opts.name) }; + }), + killSession: (opts) => + track(() => { + if (fake.failKill) { + return Promise.resolve({ exitCode: 1, stdout: "", stderr: "busy" }); + } + fake.killed.push(sessions.get(opts.name) ?? ""); + if (fake.eventsPath) { + appendFileSync(fake.eventsPath, `kill ${sessions.get(opts.name)}\n`); + } + sessions.delete(opts.name); + return Promise.resolve({ exitCode: 0, stdout: "", stderr: "" }); + }), + readLifecycleOwnerToken: (opts) => + track(async () => { + if (fake.timeoutToken) { + await Bun.sleep(TIMEOUT_DELAY_MS); + throw new MuxCommandTimeoutError(["zellij", "show"]); + } + const value = sessions.get(opts.name) ?? null; + fake.tokenReads += 1; + if (fake.tokenReads === fake.stallTokenRead) { + await Bun.sleep(STALLED_READ_MS); + } + return value; + }), + readSessionPresence: (opts) => + Promise.resolve(sessions.has(opts.name) ? "present" : "absent"), + listSessionWindowNames: () => Promise.resolve(new Set(["helper-client"])), + execInSession: () => + Promise.resolve({ exitCode: 0, stdout: "", stderr: "" }), + sendInput: () => Promise.resolve({ exitCode: 0, stdout: "", stderr: "" }), + }; + return fake; +} + +async function fixture(backendName: "zellij" | "tmux" = "zellij") { + const root = await mkdtemp(join(tmpdir(), "hack-lifecycle-ownership-")); + roots.push(root); + const projectDir = join(root, ".hack"); + await mkdir(projectDir, { recursive: true }); + const mux = fakeMux(backendName); + // A stand-in zellij for process starts (`zellij --session ... run`). + const bin = join(root, "bin"); + await mkdir(bin, { recursive: true }); + await writeFile(join(bin, "zellij"), "#!/bin/sh\nexit 0\n", { mode: 0o755 }); + // A stand-in tmux whose set-environment hangs (a stuck tmux server). + await writeFile( + join(bin, "tmux"), + '#!/bin/sh\ncase "$1" in set-environment) sleep 30 ;; esac\nexit 0\n', + { mode: 0o755 } + ); + const resolvedMux = () => + Promise.resolve({ + mode: backendName, + backends: new Map([[backendName, mux as unknown as MuxBackend]]), + }); + const ownership = createSharedHelperSessionOwnership({ + composeProject: COMPOSE_PROJECT, + hackDir: join(root, "hack-home"), + decisionTimeoutMs: 5000, + }); + const controller = (opts: { + readonly sessionOwnership: Parameters< + typeof createLifecycleProcessController + >[0]["sessionOwnership"]; + readonly prepareLaunch?: () => Promise; + readonly muxCommandTimeoutMs?: number; + readonly env?: Readonly>; + /** Receives the cleanup the controller registers for SIGINT/SIGTERM. */ + readonly onSignalCleanup?: (cleanup: () => Promise) => void; + }) => + createLifecycleProcessController({ + mux: resolvedMux, + project: { projectRoot: root, projectDir }, + projectName: "demo", + branch: "branch-a", + env: { PATH: `${bin}:${process.env.PATH ?? ""}`, ...opts.env }, + ...(opts.muxCommandTimeoutMs + ? { muxCommandTimeoutMs: opts.muxCommandTimeoutMs } + : {}), + composeProject: COMPOSE_PROJECT, + definitionHash: "sha256:test", + ...(opts.sessionOwnership + ? { sessionOwnership: opts.sessionOwnership } + : {}), + installSignalCleanup: ({ cleanup }) => { + opts.onSignalCleanup?.(cleanup); + return { dispose: () => undefined }; + }, + ...(opts.prepareLaunch + ? { + prepareLaunch: async () => { + await opts.prepareLaunch?.(); + return { command: "true", started: () => Promise.resolve() }; + }, + } + : {}), + }); + const entries = async () => + (await readLifecycleState({ projectDir })).filter( + (entry) => entry.composeProject === COMPOSE_PROJECT + ); + // hack down's stop path, against the same backend. + const stop = () => + stopLifecycleProcessController({ + project: { projectRoot: root, projectDir }, + definitionHash: "sha256:test", + projectName: "demo", + branch: "branch-a", + composeProject: COMPOSE_PROJECT, + mux: resolvedMux, + }); + const newOwnership = () => + createSharedHelperSessionOwnership({ + composeProject: COMPOSE_PROJECT, + hackDir: join(root, "hack-home"), + decisionTimeoutMs: 5000, + }); + return { + root, + bin, + projectDir, + resolvedMux, + mux, + ownership, + controller, + entries, + stop, + newOwnership, + }; +} + +const PROCESS = { name: "helper-client", command: "sleep 600" }; + +test("a failed session epoch record stops the created session before removing its entry", async () => { + const { mux, ownership, controller, entries } = await fixture(); + const starter = controller({ + sessionOwnership: { + withLock: ownership.withLock, + runIfOwned: ownership.runIfOwned, + record: (kind) => + kind === "created" + ? Promise.reject(new Error("epoch record unwritable")) + : ownership.record(kind), + }, + }); + await expect(starter.startMany({ processes: [PROCESS] })).rejects.toThrow( + "epoch record unwritable" + ); + // Never a live session without an ownership record. + expect(mux.sessions.has(SESSION)).toBe(false); + expect(await entries()).toEqual([]); + expect(starter.getOperationCleanup()).toBeNull(); +}); + +test("when that stop is not confirmed, the entry and token stay so the session is reclaimable", async () => { + const { mux, ownership, controller, entries } = await fixture(); + mux.failKill = true; + const starter = controller({ + sessionOwnership: { + withLock: ownership.withLock, + runIfOwned: ownership.runIfOwned, + record: (kind) => + kind === "created" + ? Promise.reject(new Error("epoch record unwritable")) + : ownership.record(kind), + }, + }); + await expect(starter.startMany({ processes: [PROCESS] })).rejects.toThrow( + "epoch record unwritable" + ); + const token = mux.sessions.get(SESSION); + expect(token).toBeDefined(); + const kept = await entries(); + expect(kept.map((entry) => entry.ownershipToken)).toEqual([token]); + // The caller's cleanup path (startLifecycleProcesses aborts on failure) + // must not drop the record while the stop is still unconfirmed, and must + // report the retained session so dependent helpers are kept too. + const operationCleanup = starter.getOperationCleanup(); + expect(operationCleanup).not.toBeNull(); + expect(await operationCleanup?.()).toBe("session-retained"); + // abort(), used by the start's own failure path, reports it too. + expect(await starter.abort()).toBe("session-retained"); + await starter.abort(); + expect(mux.sessions.get(SESSION)).toBe(token); + expect((await entries()).map((entry) => entry.ownershipToken)).toEqual([ + token, + ]); + // Once the session can be stopped, the kept token reclaims it. + mux.failKill = false; + await starter.abort(); + expect(mux.sessions.has(SESSION)).toBe(false); + expect(await entries()).toEqual([]); +}); + +test("a start that decided to create refuses after an overlapping start created and finalized the session", async () => { + const { projectDir, mux, ownership, controller, entries } = await fixture(); + let resume = (): void => undefined; + const paused = new Promise((resolvePause) => { + resume = resolvePause; + }); + let reachedPause = (): void => undefined; + const atPause = new Promise((resolveAt) => { + reachedPause = resolveAt; + }); + // A: decides the session is absent, then pauses before creating it. + const a = controller({ + sessionOwnership: ownership, + prepareLaunch: async () => { + reachedPause(); + await paused; + }, + }); + const aStart = a.startMany({ processes: [PROCESS] }); + await atPause; + // B: creates the session and finalizes its lifecycle entry meanwhile. + const bOwnership = createSharedHelperSessionOwnership({ + composeProject: COMPOSE_PROJECT, + hackDir: join(projectDir, "..", "hack-home"), + decisionTimeoutMs: 5000, + }); + await bOwnership.withLock(async () => { + await mux.createSession({ name: SESSION, lifecycleOwnerToken: "token-b" }); + await upsertLifecycleStateEntry({ + projectDir, + entry: { + composeProject: COMPOSE_PROJECT, + projectName: "demo", + branch: "branch-a", + sessionName: SESSION, + backend: "zellij", + ownershipToken: "token-b", + definitionHash: "sha256:test", + processes: [ + { + name: "helper-client", + windowName: "helper-client", + logPath: join(projectDir, "lifecycle.log"), + }, + ], + updatedAt: new Date().toISOString(), + }, + }); + await bOwnership.record("created"); + }); + resume(); + await expect(aStart).rejects.toThrow("changed lifecycle session"); + // B's session stays live and owned by B's persisted entry (reclaimable). + expect(mux.sessions.get(SESSION)).toBe("token-b"); + expect((await entries()).map((entry) => entry.ownershipToken)).toEqual([ + "token-b", + ]); + // A published nothing and keeps no cleanup authority over B's session. + expect(a.getOperationCleanup()).toBeNull(); + await a.abort(); + expect(mux.sessions.get(SESSION)).toBe("token-b"); +}); + +test("a start displaced after creating its session, before finalizing, leaves the replacement owned and reclaimable", async () => { + const { mux, controller, entries, stop, newOwnership } = await fixture(); + // A creates its session and starts a persistent up.before process, then + // waits in a later foreground hook (its entry is still pending). + const a = controller({ sessionOwnership: newOwnership() }); + await a.startFromCommand({ + command: { command: "sleep 600", persistent: true }, + serviceName: "persistent-hook", + }); + const tokenA = mux.sessions.get(SESSION); + expect(tokenA).toBeDefined(); + expect((await entries())[0]?.processes).toEqual([]); + // B sees an owned stale (pending) session, replaces it, creates its own + // and finalizes. + const b = controller({ sessionOwnership: newOwnership() }); + await b.startMany({ processes: [PROCESS] }); + await b.finalize(); + const tokenB = mux.sessions.get(SESSION); + expect(tokenB).toBeDefined(); + expect(tokenB).not.toBe(tokenA); + // A resumes and finalizes: it must not publish over B's entry. + await expect(a.finalize()).rejects.toBeInstanceOf( + LifecycleSessionDisplacedError + ); + expect((await entries()).map((entry) => entry.ownershipToken)).toEqual([ + tokenB, + ]); + // A gave up its authority: its abort leaves B's session alone. + expect(a.getOperationCleanup()).toBeNull(); + await a.abort(); + expect(mux.sessions.get(SESSION)).toBe(tokenB); + // hack down reclaims B through the persisted entry. + await stop(); + expect(mux.sessions.has(SESSION)).toBe(false); + expect(await entries()).toEqual([]); +}); + +test("a timed-out session creation settles before the lock is released and its outcome stays reclaimable", async () => { + const { mux, controller, entries, stop, newOwnership } = await fixture(); + // The backend killed the timed-out client after the server created the + // session; the follow-up token read times out too (outcome unknown). + mux.timeoutCreate = "created"; + mux.timeoutToken = true; + const a = controller({ sessionOwnership: newOwnership() }); + await expect(a.startMany({ processes: [PROCESS] })).rejects.toBeInstanceOf( + MuxCommandTimeoutError + ); + // Nothing of A's is still running: no completion can arrive after another + // command takes the lock. + expect(mux.inFlight).toBe(0); + const free = newOwnership(); + await expect( + Promise.race([ + free.withLock(() => Promise.resolve("acquired")), + Bun.sleep(2000).then(() => "still-held"), + ]) + ).resolves.toBe("acquired"); + // Unknown outcome: the pending entry and token are kept, matching the live + // session, so hack down reclaims it. + const kept = await entries(); + expect(kept).toHaveLength(1); + expect(mux.sessions.get(SESSION)).toBe(kept[0]?.ownershipToken); + expect(a.getOperationCleanup()).not.toBeNull(); + mux.timeoutToken = false; + await stop(); + expect(mux.sessions.has(SESSION)).toBe(false); + expect(await entries()).toEqual([]); +}); + +test("after a timed-out creation that created nothing, a later down and up see no late session", async () => { + const { mux, controller, entries, stop, newOwnership } = await fixture(); + mux.timeoutCreate = "not-created"; + const a = controller({ sessionOwnership: newOwnership() }); + await expect(a.startMany({ processes: [PROCESS] })).rejects.toBeInstanceOf( + MuxCommandTimeoutError + ); + expect(mux.inFlight).toBe(0); + // Settled and proven absent: the pending entry is retired. + expect(await entries()).toEqual([]); + mux.timeoutCreate = null; + // A down, then another up, find a consistent instance; no late creation + // of A's can appear underneath them. + await stop(); + const b = controller({ sessionOwnership: newOwnership() }); + await b.startMany({ processes: [PROCESS] }); + await b.finalize(); + const tokenB = mux.sessions.get(SESSION); + expect((await entries()).map((entry) => entry.ownershipToken)).toEqual([ + tokenB, + ]); + await a.abort(); + expect(mux.sessions.get(SESSION)).toBe(tokenB); + await stop(); + expect(mux.sessions.has(SESSION)).toBe(false); +}); + +test("a stalled replacement holds the lock until it settles, so an overlapping up never loses its session to it", async () => { + const { projectDir, mux, controller, entries, stop, newOwnership } = + await fixture(); + // A stale pending session from a crashed start. + mux.sessions.set(SESSION, "token-x"); + await upsertLifecycleStateEntry({ + projectDir, + entry: { + composeProject: COMPOSE_PROJECT, + projectName: "demo", + branch: "branch-a", + sessionName: SESSION, + backend: "zellij", + ownershipToken: "token-x", + definitionHash: "sha256:test", + processes: [], + updatedAt: new Date().toISOString(), + }, + }); + // A's replacement stalls in its kill's ownership-token read (read 2; read + // 1 is A's inspection): the read returns the token from when it started. + // B starts meanwhile. + mux.stallTokenRead = 2; + const a = controller({ sessionOwnership: newOwnership() }); + const b = controller({ sessionOwnership: newOwnership() }); + const aRun = a.startMany({ processes: [PROCESS] }).then(() => a.finalize()); + await Bun.sleep(50); + const bRun = b.startMany({ processes: [PROCESS] }).then(() => b.finalize()); + await Promise.allSettled([aRun, bRun]); + expect(mux.inFlight).toBe(0); + // Whichever start finished last owns a live session with a matching + // entry; no stale continuation killed it or removed its entry. + const live = mux.sessions.get(SESSION); + expect(live).toBeDefined(); + expect((await entries()).map((entry) => entry.ownershipToken)).toEqual([ + live, + ]); + expect(mux.killed.filter((token) => token !== "token-x")).not.toContain(live); + await stop(); + expect(mux.sessions.has(SESSION)).toBe(false); + expect(await entries()).toEqual([]); +}); + +test("a timed-out session listing is unknown, never absence: up and down leave a healthy owned session alone", async () => { + const { mux, controller, entries, stop, newOwnership } = await fixture(); + const owner = controller({ sessionOwnership: newOwnership() }); + await owner.startMany({ processes: [PROCESS] }); + await owner.finalize(); + const token = mux.sessions.get(SESSION); + const before = await entries(); + mux.timeoutList = true; + const other = controller({ sessionOwnership: newOwnership() }); + await expect( + other.startMany({ processes: [PROCESS] }) + ).rejects.toBeInstanceOf(MuxCommandTimeoutError); + await expect(stop()).rejects.toBeInstanceOf(MuxCommandTimeoutError); + expect(mux.sessions.get(SESSION)).toBe(token); + expect(await entries()).toEqual(before); + mux.timeoutList = false; + await stop(); + expect(mux.sessions.has(SESSION)).toBe(false); +}); + +test("SIGTERM during tmux environment setup settles within the mux bound", async () => { + const { bin, mux, controller, entries } = await fixture("tmux"); + const originalPath = process.env.PATH; + process.env.PATH = `${bin}:${originalPath ?? ""}`; + try { + const a = controller({ + sessionOwnership: undefined, + muxCommandTimeoutMs: 300, + }); + const start = a.startMany({ processes: [PROCESS] }).then( + () => "started", + (error: unknown) => error + ); + // Wait until the session exists and set-environment is hanging. + while (!mux.sessions.has(SESSION)) { + await Bun.sleep(10); + } + const cleanup = a.getOperationCleanup(); + expect(cleanup).not.toBeNull(); + // The signal cleanup waits for session creation to settle; it must not + // wait on the stuck tmux command forever. + const settled = await Promise.race([ + (cleanup?.() ?? Promise.resolve()).then(() => "settled"), + Bun.sleep(5000).then(() => "still waiting"), + ]); + expect(settled).toBe("settled"); + expect(await start).toBeInstanceOf(MuxCommandTimeoutError); + expect(mux.sessions.has(SESSION)).toBe(false); + expect(await entries()).toEqual([]); + } finally { + process.env.PATH = originalPath; + } +}); + +/** A stand-in zellij that logs each call and fails `run` once `fail-run` exists. */ +async function loggingZellij(bin: string): Promise { + const log = join(bin, "zellij.log"); + await writeFile( + join(bin, "zellij"), + [ + "#!/bin/sh", + `echo "$*" >> '${log}'`, + `[ -e '${join(bin, "fail-run")}' ] && exit 1`, + "exit 0", + "", + ].join("\n"), + { mode: 0o755 } + ); + return log; +} + +async function readLines(path: string): Promise { + const text = await readFile(path, "utf8").catch(() => ""); + return text.split("\n").filter((line) => line.length > 0); +} + +test("a startup whose second process fails and whose session stop is unconfirmed keeps the shared helpers (real startup catch)", async () => { + const { root, projectDir, bin, mux, resolvedMux, newOwnership, entries } = + await fixture(); + const log = await loggingZellij(bin); + mux.failKill = true; + const second = { name: "second", command: "sleep 600" }; + const startup = startLifecycleProcesses({ + project: { projectRoot: root, projectDir } as Parameters< + typeof startLifecycleProcesses + >[0]["project"], + // Only lifecycle.processes is read on this path. + cfg: { + lifecycle: { processes: [PROCESS, second] }, + } as unknown as Parameters[0]["cfg"], + projectName: "demo", + branch: "branch-a", + env: { PATH: `${bin}:${process.env.PATH ?? ""}` }, + composeProject: COMPOSE_PROJECT, + definitionHash: "sha256:test", + sessionOwnership: newOwnership(), + mux: async () => { + // The second process start fails once the first one launched. + if ((await readLines(log)).length > 0) { + await writeFile(join(bin, "fail-run"), ""); + } + return await resolvedMux(); + }, + }); + const error = await startup.then( + () => null, + (failure: unknown) => failure + ); + expect(error).toBeInstanceOf(LifecycleSessionRetainedError); + expect((error as Error).message).toContain('"second"'); + // startWithSharedHelpers' catch: the session was kept, so are the helpers. + const events: string[] = []; + await releaseAfterFailedProcessStart({ + error, + abortProcesses: () => Promise.resolve(), + rollbackHelpers: () => { + events.push("rollback"); + return Promise.resolve(); + }, + }); + expect(events).toEqual([]); + const token = mux.sessions.get(SESSION); + expect(token).toBeDefined(); + expect((await entries()).map((entry) => entry.ownershipToken)).toEqual([ + token, + ]); +}); + +test("abort's token read and kill run under the instance lock: a replacement never loses its session to them", async () => { + const { mux, controller, entries, stop, newOwnership } = await fixture(); + const a = controller({ sessionOwnership: newOwnership() }); + await a.startFromCommand({ + command: { command: "sleep 600", persistent: true }, + serviceName: "persistent-hook", + }); + const tokenA = mux.sessions.get(SESSION); + expect(tokenA).toBeDefined(); + // A's abort stalls in its ownership-token read (it returns the value read + // when it started); B starts meanwhile and would replace A's session. + mux.stallTokenRead = mux.tokenReads + 1; + const aborted = a.abort(); + await Bun.sleep(100); + const b = controller({ sessionOwnership: newOwnership() }); + const bRun = b.startMany({ processes: [PROCESS] }).then(() => b.finalize()); + await Promise.all([aborted, bRun]); + const tokenB = mux.sessions.get(SESSION); + expect(tokenB).toBeDefined(); + expect(tokenB).not.toBe(tokenA); + expect(mux.killed).not.toContain(tokenB); + expect((await entries()).map((entry) => entry.ownershipToken)).toEqual([ + tokenB, + ]); + await stop(); + expect(mux.sessions.has(SESSION)).toBe(false); +}); + +test("a displaced start never launches a process into its replacement's session", async () => { + const { bin, mux, controller, entries, newOwnership } = await fixture(); + const log = await loggingZellij(bin); + const a = controller({ sessionOwnership: newOwnership() }); + await a.startFromCommand({ + command: { command: "sleep 600", persistent: true }, + serviceName: "persistent-hook", + }); + const b = controller({ sessionOwnership: newOwnership() }); + await b.startMany({ processes: [PROCESS] }); + await b.finalize(); + const tokenB = mux.sessions.get(SESSION); + // A resumes with a later process: it must fail without touching B. + await expect( + a.startFromCommand({ + command: { command: "sleep 600" }, + serviceName: "late-a-process", + }) + ).rejects.toBeInstanceOf(LifecycleSessionDisplacedError); + expect( + (await readLines(log)).filter((line) => line.includes("late-a-process")) + ).toEqual([]); + expect(mux.sessions.get(SESSION)).toBe(tokenB); + expect((await entries()).map((entry) => entry.ownershipToken)).toEqual([ + tokenB, + ]); + expect(a.getOperationCleanup()).toBeNull(); +}); + +test("tmux environment setup finishes under the creation lock: a replacement never receives a displaced start's environment", async () => { + const { bin, mux, controller, newOwnership } = await fixture("tmux"); + const events = join(bin, "events"); + mux.eventsPath = events; + // set-environment is slow; new-window fails so no launch waits on a pane. + await writeFile( + join(bin, "tmux"), + [ + "#!/bin/sh", + 'case "$1" in', + ` set-environment) echo "env-start $4" >> '${events}'; sleep 0.4; echo "env-end $4" >> '${events}' ;;`, + " new-window) exit 1 ;;", + "esac", + "exit 0", + "", + ].join("\n"), + { mode: 0o755 } + ); + const originalPath = process.env.PATH; + process.env.PATH = `${bin}:${originalPath ?? ""}`; + try { + const a = controller({ + sessionOwnership: newOwnership(), + env: { A_MARK: "a" }, + }); + const aRun = a + .startMany({ processes: [PROCESS] }) + .catch((error: unknown) => error); + // B starts while A's environment setup is in flight; it would replace + // A's pending session. + while (!(await readLines(events)).some((line) => line.startsWith("env-"))) { + await Bun.sleep(10); + } + const b = controller({ sessionOwnership: newOwnership() }); + const bRun = b + .startMany({ processes: [] }) + .catch((error: unknown) => error); + await Promise.all([aRun, bRun]); + const lines = await readLines(events); + const aEnv = lines.flatMap((line, index) => + line.includes("A_MARK") || line.includes("PATH") ? [index] : [] + ); + const kill = lines.findIndex((line) => line.startsWith("kill ")); + // A's session was replaced only after all of A's environment setup. + expect(kill).toBeGreaterThan(-1); + expect(aEnv.every((index) => index < kill)).toBe(true); + expect(mux.inFlight).toBe(0); + } finally { + process.env.PATH = originalPath; + } +}); + +test("failed-creation cleanup reads the token and kills under the creation lock: a replacement never loses its session to it", async () => { + const { mux, ownership, controller, entries, stop, newOwnership } = + await fixture(); + let recordFailed = (): void => undefined; + const failedRecord = new Promise((resolvePromise) => { + recordFailed = resolvePromise; + }); + const a = controller({ + sessionOwnership: { + withLock: ownership.withLock, + runIfOwned: ownership.runIfOwned, + record: (kind) => { + if (kind !== "created") { + return ownership.record(kind); + } + // A's cleanup then stalls in its kill's ownership-token read (read + // 2; read 1 is the cleanup's own foreign-token check). The stalled + // read returns the value read when it started. + mux.stallTokenRead = mux.tokenReads + 2; + recordFailed(); + return Promise.reject(new Error("epoch record unwritable")); + }, + }, + }); + const aRun = a + .startMany({ processes: [PROCESS] }) + .catch((error: unknown) => error); + await failedRecord; + await Bun.sleep(100); + const b = controller({ sessionOwnership: newOwnership() }); + const bRun = b.startMany({ processes: [PROCESS] }).then(() => b.finalize()); + await Promise.all([aRun, bRun]); + const tokenB = mux.sessions.get(SESSION); + expect(tokenB).toBeDefined(); + expect(mux.killed).not.toContain(tokenB); + expect((await entries()).map((entry) => entry.ownershipToken)).toEqual([ + tokenB, + ]); + await stop(); + expect(mux.sessions.has(SESSION)).toBe(false); +}); + +test("a signal cleanup already waiting on the lock never kills a session another up adopted meanwhile", async () => { + const { mux, controller, entries, stop, newOwnership } = await fixture(); + let signalCleanup: (() => Promise) | null = null; + const a = controller({ + sessionOwnership: newOwnership(), + onSignalCleanup: (cleanup) => { + signalCleanup = cleanup; + }, + }); + await a.startMany({ processes: [PROCESS] }); + await a.finalize(); + const tokenA = mux.sessions.get(SESSION); + expect(tokenA).toBeDefined(); + // B holds the instance lock while it adopts A's finalized session; A's + // signal callback starts meanwhile and waits on that lock. + const b = newOwnership(); + let release = (): void => undefined; + const gate = new Promise((resolvePromise) => { + release = resolvePromise; + }); + let entered = (): void => undefined; + const inLock = new Promise((resolvePromise) => { + entered = resolvePromise; + }); + const adoption = b.withLock(async () => { + entered(); + await gate; + await b.record("adopted"); + }); + await inLock; + expect(signalCleanup).not.toBeNull(); + const signalled = (signalCleanup as unknown as () => Promise)(); + await Bun.sleep(100); + release(); + await adoption; + expect(await signalled).toBeUndefined(); + // The adopter's session and entry are intact. + expect(mux.sessions.get(SESSION)).toBe(tokenA); + expect(mux.killed).not.toContain(tokenA); + expect((await entries()).map((entry) => entry.ownershipToken)).toEqual([ + tokenA, + ]); + await stop(); + expect(mux.sessions.has(SESSION)).toBe(false); +}); + +test("the shared-helper backend requirement holds at the backend the controller actually selects, not only at preflight", async () => { + const { mux, controller, entries, newOwnership, resolvedMux } = + await fixture(); + // Preflight sees a backend that records its token with creation. + await assertSharedHelperSessionMuxSupported({ + sharedHelpersConfigured: true, + needsSession: true, + mux: resolvedMux, + }); + // The mux selection changes while hooks run (for example sessions.mux is + // switched to zellij): the backend now selected writes its token later. + mux.createsWithOwnerToken = false; + const a = controller({ sessionOwnership: newOwnership() }); + await expect(a.startMany({ processes: [PROCESS] })).rejects.toThrow( + "need tmux" + ); + expect(mux.sessions.size).toBe(0); + expect(await entries()).toEqual([]); +}); diff --git a/tests/project-lifecycle-sessions.test.ts b/tests/project-lifecycle-sessions.test.ts index b707dd234..6a85e564e 100644 --- a/tests/project-lifecycle-sessions.test.ts +++ b/tests/project-lifecycle-sessions.test.ts @@ -6,6 +6,7 @@ import { inspectLifecycleSession, killInspectedLifecycleSession, killLifecycleSessionWithOwnership, + LifecycleSessionPresenceUnknownError, resolveLifecycleDefinitionHash, resolveLifecycleEnvironmentFingerprint, } from "../src/lib/project-lifecycle-sessions.ts"; @@ -251,15 +252,23 @@ test("inspection confirms an unlisted owned session instead of recreating it (HA }); test("inspection treats an unlisted session as absent only on explicit absence", async () => { - for (const presence of ["absent", "unknown"] as const) { - const backend = inspectionBackend({ + const absent = await inspectFixture( + inspectionBackend({ listSessions: async () => [], - readSessionPresence: async () => presence, - }); - const inspection = await inspectFixture(backend); - expect(inspection.classification).toBe("absent"); - expect(inspection.decision).toEqual({ kind: "create" }); - } + readSessionPresence: async () => "absent", + }) + ); + expect(absent.classification).toBe("absent"); + expect(absent.decision).toEqual({ kind: "create" }); + // An unknown exact-presence answer blocks: never absence, never "create". + await expect( + inspectFixture( + inspectionBackend({ + listSessions: async () => [], + readSessionPresence: async () => "unknown", + }) + ) + ).rejects.toBeInstanceOf(LifecycleSessionPresenceUnknownError); }); test("inspection never grants legacy ownership to an unlisted session", async () => { diff --git a/tests/project-restart-repair-guard.test.ts b/tests/project-restart-repair-guard.test.ts new file mode 100644 index 000000000..b18d47493 --- /dev/null +++ b/tests/project-restart-repair-guard.test.ts @@ -0,0 +1,111 @@ +import { afterEach, expect, test } from "bun:test"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { repairFailedRestart } from "../src/commands/project.ts"; +import { + DEFAULT_COMPOSE_IMAGE_TIMEOUT_MS, + DEFAULT_COMPOSE_STARTUP_TIMEOUT_MS, +} from "../src/lib/compose-startup-budget.ts"; +import { SharedHelperSupersededError } from "../src/lib/project-shared-helpers.ts"; +import { setLoggerBackendOverride } from "../src/ui/logger.ts"; + +/** + * The restart repair attempt launches containers too, so it runs behind the + * same shared helper guard as the first attempt: a concurrent down that + * released the helpers stops the repair before any container starts. + */ + +const roots: string[] = []; + +afterEach(async () => { + setLoggerBackendOverride({ backend: null }); + for (const root of roots.splice(0)) { + await rm(root, { recursive: true, force: true }); + } +}); + +const budgets = { + startupTimeoutMs: DEFAULT_COMPOSE_STARTUP_TIMEOUT_MS, + imageTimeoutMs: DEFAULT_COMPOSE_IMAGE_TIMEOUT_MS, +}; + +async function scratch(): Promise { + const root = await mkdtemp(join(tmpdir(), "hack-restart-repair-")); + roots.push(root); + return root; +} + +test("a superseded restart repair cleans up and starts no container", async () => { + const projectDir = await scratch(); + const events: string[] = []; + const result = await repairFailedRestart({ + budgets, + upCode: 1, + composePhase: "start", + assertCurrent: () => Promise.reject(new SharedHelperSupersededError()), + cleanup: () => { + events.push("cleanup"); + return Promise.resolve(); + }, + projectDir, + lifecycleComposeProject: "demo--branch-a", + up: async (beforeStart) => { + events.push(beforeStart ? "up-guarded" : "up-unguarded"); + // The backend awaits beforeStart right before containers start. + await beforeStart?.(); + events.push("containers-started"); + return 0; + }, + }); + expect(result).toEqual({ code: 1, phase: "lifecycle" }); + expect(events).toEqual(["up-guarded", "cleanup"]); +}); + +test("a current restart repair runs guarded and reports the original failure", async () => { + const projectDir = await scratch(); + const checks: unknown[] = []; + const result = await repairFailedRestart({ + budgets, + upCode: 3, + composePhase: "start", + assertCurrent: (check) => { + checks.push(check ?? "before-start"); + return Promise.resolve(); + }, + cleanup: () => Promise.reject(new Error("must not clean up a repair")), + projectDir, + lifecycleComposeProject: "demo--branch-a", + up: async (beforeStart) => { + await beforeStart?.(); + return 0; + }, + }); + expect(result).toEqual({ code: 3, phase: "compose", composePhase: "start" }); + expect(checks).toEqual(["before-start", { servicesStarted: true }]); +}); + +test("a failed repair keeps the shared helpers when containers may run, before cleanup", async () => { + const projectDir = await scratch(); + const events: string[] = []; + const result = await repairFailedRestart({ + budgets, + upCode: 1, + composePhase: "start", + assertCurrent: () => Promise.resolve(), + cleanup: () => { + events.push("cleanup"); + return Promise.resolve(); + }, + retainHelpersIfContainersMayRun: () => { + events.push("retain-check"); + return Promise.resolve(); + }, + projectDir, + lifecycleComposeProject: "demo--branch-a", + up: () => Promise.resolve(124), + }); + expect(result.code).toBe(1); + expect(events).toEqual(["retain-check", "cleanup"]); +}); diff --git a/tests/project-shared-helpers.test.ts b/tests/project-shared-helpers.test.ts new file mode 100644 index 000000000..881abbff0 --- /dev/null +++ b/tests/project-shared-helpers.test.ts @@ -0,0 +1,1388 @@ +import { afterEach, expect, test } from "bun:test"; +import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { + readProcessIdentities, + selectProcessTree, + signalVerifiedProcessTree, +} from "../src/lib/process-tree.ts"; +import { readProjectConfig } from "../src/lib/project.ts"; +import { + buildSharedHelperSpec, + captureSharedHelperShutdownGeneration, + createSharedHelperSessionOwnership, + markSharedHelperInstanceStopping, + type ProjectSharedHelperConfig, + parseProjectSharedHelpers, + releaseProjectSharedHelpers, + resolveSharedHelperHolder, + resolveSharedHelperStateDir, + SharedHelperStartError, + SharedHelperSupersededError, + startProjectSharedHelpers, +} from "../src/lib/project-shared-helpers.ts"; +import { withProjectsRegistryLock } from "../src/lib/projects-registry-lock.ts"; +import { + isSharedHostProcessRunning, + parseSharedHostClaim, +} from "../src/lib/shared-host-lifecycle.ts"; + +/** + * Project wiring for lifecycle.sharedHelpers. Two Compose projects + * (`demo--branch-a`, `demo--branch-b`) stand in for two branches; the helper + * is the portable fixture from tests/fixtures/shared-host-helper.ts. The CLI + * path (`hack up`/`down` on two branches) is the e2e scenario + * tests/e2e/scenarios/shared-host-helpers.ts. + */ + +const HELPER = resolve(import.meta.dir, "fixtures/shared-host-helper.ts"); +const TIMEOUT = 30_000; +const BRANCH_A = "demo--branch-a"; +const BRANCH_B = "demo--branch-b"; +const UPSTREAM_A = "https://issuer.branch-a.test"; + +const roots: string[] = []; +const foreign: Bun.Subprocess<"ignore", "pipe", "ignore">[] = []; + +afterEach(async () => { + for (const proc of foreign.splice(0)) { + proc.kill("SIGKILL"); + await proc.exited; + } + // Reap any owned helper through birth-verified primitives, not the code + // under test, so a failing run cannot leak fixture processes. + const census = await readProcessIdentities(); + for (const root of roots) { + for (const name of ["issuer-bridge", "second-bridge"]) { + for (const claimPath of await claimFiles(root, name)) { + const text = await readFile(claimPath, "utf8").catch(() => null); + const claim = text === null ? null : parseSharedHostClaim(text); + if ( + claim?.ownership === "owned" && + census.some( + (row) => row.pid === claim.pid && row.birth === claim.birth + ) + ) { + await signalVerifiedProcessTree( + selectProcessTree(census, claim.pid), + "SIGKILL" + ); + } + } + } + } + for (const root of roots.splice(0)) { + await rm(root, { recursive: true, force: true }); + } +}); + +async function claimFiles(root: string, name: string): Promise { + const dir = join(root, "hack-home", "shared-helpers"); + const entries = await import("node:fs/promises") + .then((fs) => fs.readdir(dir)) + .catch(() => [] as string[]); + return entries + .filter((entry) => entry.startsWith(`${name}-`)) + .map((entry) => join(dir, entry, `${name}.claim.json`)); +} + +async function scratch(): Promise { + const root = await mkdtemp(join(tmpdir(), "hack-project-shared-")); + roots.push(root); + return root; +} + +function freePort(): number { + const probe = Bun.listen({ + hostname: "127.0.0.1", + port: 0, + socket: { + data() { + // Bind probe only. + }, + }, + }); + const port = probe.port; + probe.stop(true); + return port; +} + +function helperConfig(opts: { + readonly port: number; + readonly name?: string; + readonly mode?: "identity" | "placeholder"; + readonly upstream?: string; + readonly readyTimeoutMs?: number; +}): ProjectSharedHelperConfig { + const name = opts.name ?? "issuer-bridge"; + const upstream = opts.upstream ?? UPSTREAM_A; + return { + name, + enabled: true, + command: [ + process.execPath, + HELPER, + opts.mode ?? "identity", + String(opts.port), + name, + upstream, + ], + port: opts.port, + identityPath: "/__hack/identity", + upstream, + env: ["HOME"], + readyTimeoutMs: opts.readyTimeoutMs ?? 10_000, + }; +} + +function context(root: string) { + return { + projectRoot: root, + env: {}, + hackDir: join(root, "hack-home"), + // HOME is the scratch root: the fixture exits once it is removed. + processEnv: { PATH: process.env.PATH ?? "/usr/bin:/bin", HOME: root }, + }; +} + +async function readHelperClaim( + root: string, + helper: ProjectSharedHelperConfig +) { + const text = await readFile( + join( + resolveSharedHelperStateDir({ + hackDir: join(root, "hack-home"), + name: helper.name, + port: helper.port, + }), + `${helper.name}.claim.json` + ), + "utf8" + ).catch(() => null); + return text === null ? null : parseSharedHostClaim(text); +} + +async function ownedIdentity(root: string, helper: ProjectSharedHelperConfig) { + const claim = await readHelperClaim(root, helper); + if (claim?.ownership !== "owned") { + throw new Error("expected an owned claim"); + } + return { pid: claim.pid, birth: claim.birth }; +} + +async function isAlive(identity: { pid: number; birth: string }) { + return isSharedHostProcessRunning(await readProcessIdentities(), identity); +} + +async function startForeign(opts: { + readonly root: string; + readonly mode: "identity" | "tcp"; + readonly port: number; + readonly upstream?: string; +}) { + const proc = Bun.spawn( + [ + process.execPath, + HELPER, + opts.mode, + String(opts.port), + "issuer-bridge", + opts.upstream ?? UPSTREAM_A, + ], + { + env: { PATH: process.env.PATH ?? "/usr/bin:/bin", HOME: opts.root }, + stdin: "ignore", + stdout: "pipe", + stderr: "ignore", + } + ); + foreign.push(proc); + const reader = proc.stdout.getReader(); + let text = ""; + while (!text.includes("listening")) { + const { done, value } = await reader.read(); + if (done) { + throw new Error("foreign fixture exited before listening"); + } + text += new TextDecoder().decode(value); + } + reader.releaseLock(); + const identity = (await readProcessIdentities()).find( + (row) => row.pid === proc.pid + ); + if (!identity) { + throw new Error("foreign fixture not in census"); + } + return { proc, identity: { pid: identity.pid, birth: identity.birth } }; +} + +async function served(port: number): Promise { + return await (await fetch(`http://127.0.0.1:${port}/`)).text(); +} + +test("parses a minimal shared helper with defaults", () => { + expect( + parseProjectSharedHelpers([ + { + name: "issuer-bridge", + command: ["bun", "run", "bridge"], + port: 4100, + upstreamEnv: "ISSUER_UPSTREAM", + }, + ]) + ).toEqual({ + helpers: [ + { + name: "issuer-bridge", + enabled: true, + command: ["bun", "run", "bridge"], + port: 4100, + identityPath: "/__hack/identity", + upstreamEnv: "ISSUER_UPSTREAM", + env: [], + readyTimeoutMs: 30_000, + }, + ], + errors: [], + }); + expect(parseProjectSharedHelpers(undefined)).toEqual({ errors: [] }); +}); + +test("refuses invalid shared helper entries instead of skipping them", () => { + const base = { + name: "issuer-bridge", + command: ["bridge"], + port: 4100, + upstream: "https://issuer.test", + }; + const cases: readonly [unknown, string][] = [ + [{}, "must be an array"], + [[{ ...base, command: "bun run bridge" }], "argv, not a shell string"], + [[{ ...base, upstreamEnv: "X" }], "exactly one of upstream"], + [[{ ...base, upstream: undefined }], "exactly one of upstream"], + [[{ ...base, singleton: { ports: [1] } }], 'unknown key "singleton"'], + [[{ ...base, name: "../escape" }], ".name must match"], + [[{ ...base, port: 70_000 }], ".port must be"], + [[{ ...base, env: ["NOT-A-NAME"] }], ".env must be"], + [[{ ...base, identityPath: "identity" }], ".identityPath must be"], + [[{ ...base, readyTimeoutMs: 0 }], ".readyTimeoutMs must be"], + [[base, { ...base, port: 4101 }], 'duplicates "issuer-bridge"'], + [[base, { ...base, name: "other" }], ".port duplicates 4100"], + ]; + for (const [value, message] of cases) { + const parsed = parseProjectSharedHelpers(value); + expect(parsed.helpers).toBeUndefined(); + expect(parsed.errors.join(" ")).toContain(message); + } +}); + +test("an invalid sharedHelpers entry is a project config parse error", async () => { + const root = await scratch(); + await mkdir(join(root, ".hack"), { recursive: true }); + await writeFile( + join(root, ".hack", "hack.config.json"), + JSON.stringify({ + name: "demo", + lifecycle: { + sharedHelpers: [{ name: "issuer-bridge", command: "bridge", port: 1 }], + }, + }) + ); + const cfg = await readProjectConfig({ + projectRoot: root, + projectDir: join(root, ".hack"), + } as Parameters[0]); + expect(cfg.parseError).toContain("lifecycle.sharedHelpers[0].command"); + expect(cfg.lifecycle?.sharedHelpers).toBeUndefined(); +}); + +test("the claim directory depends on helper and port only; the holder is the compose project", () => { + const dir = resolveSharedHelperStateDir({ + hackDir: "/h", + name: "issuer-bridge", + port: 4100, + }); + expect(dir).toBe("/h/shared-helpers/issuer-bridge-4100"); + expect(resolveSharedHelperHolder(BRANCH_A)).toBe(BRANCH_A); + expect(() => resolveSharedHelperHolder("demo/branch a")).toThrow( + "Compose project name" + ); +}); + +test("the helper gets PATH and only its listed env; upstreamEnv reads lifecycle env first", () => { + const spec = buildSharedHelperSpec({ + helper: { + ...helperConfig({ port: 4100 }), + upstream: undefined, + upstreamEnv: "ISSUER_UPSTREAM", + env: ["ISSUER_TOKEN"], + }, + projectRoot: "/repo", + env: { ISSUER_UPSTREAM: "https://lifecycle.test", ISSUER_TOKEN: "t" }, + processEnv: { + PATH: "/bin", + ISSUER_UPSTREAM: "https://process.test", + RUNNER_ONLY: "x", + }, + hackDir: "/h", + }); + expect(spec.env).toEqual({ PATH: "/bin", ISSUER_TOKEN: "t" }); + expect(spec.expected).toEqual({ + helper: "issuer-bridge", + upstream: "https://lifecycle.test", + }); + expect(spec.stateDir).toBe("/h/shared-helpers/issuer-bridge-4100"); + expect(() => + buildSharedHelperSpec({ + helper: { ...helperConfig({ port: 4100 }), env: ["MISSING_NAME"] }, + projectRoot: "/repo", + env: {}, + processEnv: { PATH: "/bin" }, + }) + ).toThrow("lists env MISSING_NAME, but it is not set"); +}); + +test( + "two branches share one owned helper; the last branch release stops it", + async () => { + const root = await scratch(); + const helper = helperConfig({ port: freePort() }); + const a = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + expect(a.started[0]?.result).toMatchObject({ + state: "ready", + ownership: "owned", + spawned: true, + holderAdded: true, + }); + const owned = await ownedIdentity(root, helper); + const b = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_B, + }); + expect(b.started[0]?.result).toMatchObject({ + state: "ready", + ownership: "owned", + spawned: false, + pid: owned.pid, + }); + expect(await served(helper.port)).toBe(`served:${UPSTREAM_A}`); + expect((await readHelperClaim(root, helper))?.holders).toEqual([ + BRANCH_A, + BRANCH_B, + ]); + + const releaseA = await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir: join(root, "hack-home"), + }); + expect(releaseA.map((entry) => entry.result.state)).toEqual(["retained"]); + expect(await isAlive(owned)).toBe(true); + expect(await served(helper.port)).toBe(`served:${UPSTREAM_A}`); + + const releaseB = await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_B, + hackDir: join(root, "hack-home"), + }); + expect(releaseB.map((entry) => entry.result.state)).toEqual(["stopped"]); + expect(await isAlive(owned)).toBe(false); + expect(await readHelperClaim(root, helper)).toBeNull(); + }, + TIMEOUT +); + +test( + "a later helper's failure rolls back only the holders this run added; foreign listeners survive", + async () => { + const root = await scratch(); + const first = helperConfig({ port: freePort() }); + const second = helperConfig({ + port: freePort(), + name: "second-bridge", + }); + // The second port is held by a different helper kind (foreign). + const other = await startForeign({ + root, + mode: "identity", + port: second.port, + }); + + // Branch A already holds the first helper from an earlier successful up. + await startProjectSharedHelpers({ + ...context(root), + helpers: [first], + composeProject: BRANCH_A, + }); + const owned = await ownedIdentity(root, first); + + // Branch B's up acquires the first helper, then the second is refused: + // B's new holder on the first helper is rolled back. + const failedUp = startProjectSharedHelpers({ + ...context(root), + helpers: [first, second], + composeProject: BRANCH_B, + }); + await expect(failedUp).rejects.toBeInstanceOf(SharedHelperStartError); + await expect(failedUp).rejects.toThrow("(foreign-listener)"); + expect((await readHelperClaim(root, first))?.holders).toEqual([BRANCH_A]); + expect(await isAlive(owned)).toBe(true); + expect(await isAlive(other.identity)).toBe(true); + + // Branch A re-runs up with the same failure: its rollback must not drop + // the holder its earlier up recorded, so the helper keeps running. + await expect( + startProjectSharedHelpers({ + ...context(root), + helpers: [first, second], + composeProject: BRANCH_A, + }) + ).rejects.toThrow("(foreign-listener)"); + expect((await readHelperClaim(root, first))?.holders).toEqual([BRANCH_A]); + expect(await isAlive(owned)).toBe(true); + expect(await isAlive(other.identity)).toBe(true); + expect(await served(first.port)).toBe(`served:${UPSTREAM_A}`); + }, + TIMEOUT +); + +test( + "wrong upstream and a foreign port fail the start, spawn nothing and leave the listener running", + async () => { + const root = await scratch(); + const wrongPort = freePort(); + const wrong = await startForeign({ + root, + mode: "identity", + port: wrongPort, + upstream: "https://issuer.branch-b.test", + }); + await expect( + startProjectSharedHelpers({ + ...context(root), + helpers: [helperConfig({ port: wrongPort })], + composeProject: BRANCH_A, + }) + ).rejects.toThrow("(wrong-upstream)"); + expect(await isAlive(wrong.identity)).toBe(true); + expect(await served(wrongPort)).toBe("served:https://issuer.branch-b.test"); + + const tcpPort = freePort(); + const tcp = await startForeign({ root, mode: "tcp", port: tcpPort }); + const error = await startProjectSharedHelpers({ + ...context(root), + helpers: [helperConfig({ port: tcpPort, readyTimeoutMs: 3000 })], + composeProject: BRANCH_A, + }).catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(SharedHelperStartError); + expect(String(error)).toContain("(foreign-listener)"); + expect(String(error)).not.toContain("branch-b.test"); + expect(await isAlive(tcp.identity)).toBe(true); + expect( + await readHelperClaim(root, helperConfig({ port: tcpPort })) + ).toBeNull(); + }, + TIMEOUT +); + +test( + "a healthy helper started elsewhere is adopted and never stopped by down", + async () => { + const root = await scratch(); + const port = freePort(); + const existing = await startForeign({ root, mode: "identity", port }); + const helper = helperConfig({ port }); + const up = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + expect(up.started[0]?.result).toMatchObject({ + state: "ready", + ownership: "adopted", + }); + const down = await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir: join(root, "hack-home"), + }); + expect(down.map((entry) => entry.result.state)).toEqual([ + "released-adopted", + ]); + expect(await isAlive(existing.identity)).toBe(true); + expect(await served(port)).toBe(`served:${UPSTREAM_A}`); + }, + TIMEOUT +); + +test( + "shutdown during readiness retry cancels, stops the owned child and leaves no claim", + async () => { + const root = await scratch(); + const helper = helperConfig({ + port: freePort(), + mode: "placeholder", + readyTimeoutMs: 20_000, + }); + const controller = new AbortController(); + const pending = startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + signal: controller.signal, + }); + let child: { pid: number; birth: string } | null = null; + const deadline = performance.now() + 10_000; + while (child === null && performance.now() < deadline) { + const claim = await readHelperClaim(root, helper); + if (claim?.ownership === "owned") { + child = { pid: claim.pid, birth: claim.birth }; + } + await Bun.sleep(50); + } + if (child === null) { + throw new Error("the placeholder was never claimed"); + } + await Bun.sleep(300); + controller.abort(); + await expect(pending).rejects.toThrow("(cancelled)"); + expect(await isAlive(child)).toBe(false); + expect(await readHelperClaim(root, helper)).toBeNull(); + }, + TIMEOUT +); + +test("a disabled helper starts nothing and down still releases a recorded holder", async () => { + const root = await scratch(); + const helper = { ...helperConfig({ port: freePort() }), enabled: false }; + const up = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + expect(up.started[0]?.result).toEqual({ + state: "disabled", + ownership: "disabled", + }); + expect(await readHelperClaim(root, helper)).toBeNull(); + const down = await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir: join(root, "hack-home"), + }); + expect(down.map((entry) => entry.result.state)).toEqual(["absent"]); +}); + +test( + "an overlapping start of the same instance is never revoked by the earlier run's rollback", + async () => { + const root = await scratch(); + const helper = helperConfig({ port: freePort() }); + // Run 1 adds the holder; run 2 of the same instance reuses it. + const first = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + const owned = await ownedIdentity(root, helper); + const second = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + expect(second.started[0]?.result).toMatchObject({ + state: "ready", + holderAdded: false, + }); + // Run 1 then fails later (Compose): its rollback must not stop the + // helper under run 2. + expect(await first.rollback()).toEqual([]); + expect(await isAlive(owned)).toBe(true); + expect((await readHelperClaim(root, helper))?.holders).toEqual([BRANCH_A]); + expect(await served(helper.port)).toBe(`served:${UPSTREAM_A}`); + // Without a newer start, a rollback releases what its run added. + await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir: join(root, "hack-home"), + }); + const fresh = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + const freshOwned = await ownedIdentity(root, helper); + expect((await fresh.rollback()).map((entry) => entry.result.state)).toEqual( + ["stopped"] + ); + expect(await isAlive(freshOwned)).toBe(false); + }, + TIMEOUT +); + +test("a disabled helper needs none of its runtime env", async () => { + const root = await scratch(); + const helper: ProjectSharedHelperConfig = { + ...helperConfig({ port: freePort() }), + enabled: false, + upstream: undefined, + upstreamEnv: "UNSET_ISSUER_UPSTREAM", + env: ["UNSET_ISSUER_TOKEN"], + }; + const up = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + expect(up.started[0]?.result).toEqual({ + state: "disabled", + ownership: "disabled", + }); +}); + +test( + "a signal anywhere in a guarded start runs its cleanup, then is re-raised", + async () => { + const root = await scratch(); + const script = join(root, "guard.ts"); + const marker = join(root, "cleaned"); + const module = resolve( + import.meta.dir, + "../src/lib/project-shared-helpers.ts" + ); + await writeFile( + script, + `import { writeFileSync } from "node:fs"; +import { withTerminationAbort } from ${JSON.stringify(module)}; +await withTerminationAbort( + async () => { + // A step that does not watch the abort signal, like mux session setup. + console.log("running"); + await Bun.sleep(1500); + }, + { onSignalled: async () => writeFileSync(${JSON.stringify(marker)}, "yes") } +); +console.log("not reached"); +` + ); + const proc = Bun.spawn([process.execPath, script], { + stdin: "ignore", + stdout: "pipe", + stderr: "ignore", + }); + const reader = proc.stdout.getReader(); + let text = ""; + while (!text.includes("running")) { + const { done, value } = await reader.read(); + if (done) { + break; + } + text += new TextDecoder().decode(value); + } + reader.releaseLock(); + proc.kill("SIGTERM"); + expect(await proc.exited).toBe(143); + expect(await readFile(marker, "utf8")).toBe("yes"); + }, + TIMEOUT +); + +test( + "a release waits for a start of the same instance, so up and down never split the helper set", + async () => { + const root = await scratch(); + const first = helperConfig({ port: freePort() }); + const second = helperConfig({ port: freePort(), name: "second-bridge" }); + // Pause the start between its two helpers (the report hook is awaited). + let resume = (): void => undefined; + const paused = new Promise((resolvePause) => { + resume = resolvePause; + }); + let reachedPause = (): void => undefined; + const atPause = new Promise((resolveAt) => { + reachedPause = resolveAt; + }); + const start = startProjectSharedHelpers({ + ...context(root), + helpers: [first, second], + composeProject: BRANCH_A, + report: async (record) => { + if (record.name === first.name) { + reachedPause(); + await paused; + } + }, + }); + await atPause; + const firstHelper = await ownedIdentity(root, first); + let released = false; + const release = releaseProjectSharedHelpers({ + helpers: [first, second], + composeProject: BRANCH_A, + hackDir: join(root, "hack-home"), + }).then((result) => { + released = true; + return result; + }); + await Bun.sleep(500); + // The release must not run inside the start: it waits for the instance. + expect(released).toBe(false); + expect(await isAlive(firstHelper)).toBe(true); + resume(); + await start; + const secondHelper = await ownedIdentity(root, second); + expect((await release).map((entry) => entry.result.state).sort()).toEqual([ + "stopped", + "stopped", + ]); + // Never split: both helpers end up stopped together. + expect(await isAlive(firstHelper)).toBe(false); + expect(await isAlive(secondHelper)).toBe(false); + }, + TIMEOUT +); + +test( + "a down that marks the instance stopping, or releases it, supersedes an in-flight start", + async () => { + const root = await scratch(); + const helper = helperConfig({ port: freePort() }); + const hackDir = join(root, "hack-home"); + const up = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + await up.assertCurrent(); + expect( + await markSharedHelperInstanceStopping({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir, + }) + ).toBe(true); + const error = await up.assertCurrent().catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(SharedHelperSupersededError); + expect(String(error)).toContain("before launching services"); + // The superseded start's rollback must not release: the down owns it. + expect(await up.rollback()).toEqual([]); + + // A release without a prior mark (down that skipped marking) also + // supersedes, and the after-start check names started services. + const again = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir, + }); + const after = await again + .assertCurrent({ servicesStarted: true }) + .catch((caught: unknown) => caught); + expect(after).toBeInstanceOf(SharedHelperSupersededError); + expect(String(after)).toContain("while its services started"); + expect(await readHelperClaim(root, helper)).toBeNull(); + }, + TIMEOUT +); + +test( + "a down interrupted between its generation and operation-record writes still keeps a post-start rollback from releasing", + async () => { + const root = await scratch(); + const helper = helperConfig({ port: freePort() }); + const hackDir = join(root, "hack-home"); + const up = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + await up.assertCurrent(); + const owned = await ownedIdentity(root, helper); + // The containers started; then a down published only its new shutdown + // generation and died before replacing the operation record. + const instanceDir = join(hackDir, "shared-helpers", ".instances"); + const holder = resolveSharedHelperHolder(BRANCH_A); + await writeFile( + join(instanceDir, `${holder}.shutdown-generation`), + "interrupted-down\n" + ); + const error = await up + .assertCurrent({ servicesStarted: true }) + .catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(SharedHelperSupersededError); + // The superseded start's cleanup keeps the helper under its containers. + expect(await up.rollback()).toEqual([]); + expect(await isAlive(owned)).toBe(true); + expect((await readHelperClaim(root, helper))?.holders).toEqual([holder]); + // The next down releases it. + await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir, + }); + expect(await isAlive(owned)).toBe(false); + expect(await readHelperClaim(root, helper)).toBeNull(); + }, + TIMEOUT +); + +test( + "down releases helpers the instance acquired even after their config entry is removed or changed", + async () => { + const root = await scratch(); + const hackDir = join(root, "hack-home"); + const original = helperConfig({ port: freePort() }); + await startProjectSharedHelpers({ + ...context(root), + helpers: [original], + composeProject: BRANCH_A, + }); + const owned = await ownedIdentity(root, original); + // The entry is removed from config: down still releases and stops it. + const removed = await releaseProjectSharedHelpers({ + helpers: undefined, + composeProject: BRANCH_A, + hackDir, + }); + expect(removed.map((entry) => [entry.name, entry.result.state])).toEqual([ + ["issuer-bridge", "stopped"], + ]); + expect(await isAlive(owned)).toBe(false); + expect(await readHelperClaim(root, original)).toBeNull(); + + // The entry's port changes after start: the original port is released. + await startProjectSharedHelpers({ + ...context(root), + helpers: [original], + composeProject: BRANCH_A, + }); + const ownedAgain = await ownedIdentity(root, original); + const moved = helperConfig({ port: freePort() }); + const changed = await releaseProjectSharedHelpers({ + helpers: [moved], + composeProject: BRANCH_A, + hackDir, + }); + expect(changed.map((entry) => entry.result.state).sort()).toEqual([ + "absent", + "stopped", + ]); + expect(await isAlive(ownedAgain)).toBe(false); + expect(await readHelperClaim(root, original)).toBeNull(); + // Nothing is left to release. + expect( + await releaseProjectSharedHelpers({ + helpers: undefined, + composeProject: BRANCH_A, + hackDir, + }) + ).toEqual([]); + }, + TIMEOUT +); + +test( + "a failed release keeps the claim and the acquired record, so the next down retries it", + async () => { + const root = await scratch(); + const hackDir = join(root, "hack-home"); + const helper = helperConfig({ port: freePort() }); + await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + const owned = await ownedIdentity(root, helper); + const claimPath = join( + resolveSharedHelperStateDir({ + hackDir, + name: helper.name, + port: helper.port, + }), + `${helper.name}.claim.json` + ); + const claimText = await readFile(claimPath, "utf8"); + await writeFile(claimPath, "{not json\n", { mode: 0o600 }); + // Config entry removed too: the record alone must carry the retry. + const failed = await releaseProjectSharedHelpers({ + helpers: undefined, + composeProject: BRANCH_A, + hackDir, + }); + expect(failed.map((entry) => entry.result)).toMatchObject([ + { state: "failed", reason: "claim-invalid" }, + ]); + expect(await readFile(claimPath, "utf8")).toBe("{not json\n"); + expect(await isAlive(owned)).toBe(true); + await writeFile(claimPath, claimText, { mode: 0o600 }); + const retried = await releaseProjectSharedHelpers({ + helpers: undefined, + composeProject: BRANCH_A, + hackDir, + }); + expect(retried.map((entry) => entry.result.state)).toEqual(["stopped"]); + expect(await isAlive(owned)).toBe(false); + }, + TIMEOUT +); + +test( + "a newer start of the same instance does not supersede an earlier one; a down supersedes both", + async () => { + const root = await scratch(); + const helper = helperConfig({ port: freePort() }); + const first = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + const second = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + // The newer start holds the helpers: the earlier start may still launch. + await first.assertCurrent(); + await first.assertCurrent({ servicesStarted: true }); + await second.assertCurrent(); + expect( + await markSharedHelperInstanceStopping({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir: join(root, "hack-home"), + }) + ).toBe(true); + for (const start of [first, second]) { + expect( + await start.assertCurrent().catch((caught: unknown) => caught) + ).toBeInstanceOf(SharedHelperSupersededError); + } + await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir: join(root, "hack-home"), + }); + }, + TIMEOUT +); + +test( + "with no helper configured, an unreadable record or a held lock still fails the release", + async () => { + const root = await scratch(); + const hackDir = join(root, "hack-home"); + const instanceDir = join(hackDir, "shared-helpers", ".instances"); + await mkdir(instanceDir, { recursive: true, mode: 0o700 }); + const record = join(instanceDir, `${BRANCH_A}.acquired.json`); + await writeFile(record, "{not json\n", { mode: 0o600 }); + // Both "no sharedHelpers key" and an empty list must report it. + for (const helpers of [undefined, []]) { + const failed = await releaseProjectSharedHelpers({ + helpers, + composeProject: BRANCH_A, + hackDir, + }); + expect(failed.map((entry) => entry.name)).toEqual(["recorded-helpers"]); + } + const unreadable = await releaseProjectSharedHelpers({ + helpers: undefined, + composeProject: BRANCH_A, + hackDir, + }); + expect(unreadable.map((entry) => [entry.name, entry.result])).toEqual([ + [ + "recorded-helpers", + expect.objectContaining({ + state: "failed", + reason: "state-unavailable", + }), + ], + ]); + expect(await readFile(record, "utf8")).toBe("{not json\n"); + + await writeFile( + record, + `${JSON.stringify([{ name: "issuer-bridge", port: 4100 }])}\n`, + { mode: 0o600 } + ); + let releaseLock = (): void => undefined; + const held = new Promise((resolveHeld) => { + releaseLock = resolveHeld; + }); + let acquired = (): void => undefined; + const isHeld = new Promise((resolveAcquired) => { + acquired = resolveAcquired; + }); + const holder = withProjectsRegistryLock({ + lockPath: join(instanceDir, `${BRANCH_A}.lock`), + run: async () => { + acquired(); + await held; + }, + }); + await isHeld; + const locked = await releaseProjectSharedHelpers({ + helpers: undefined, + composeProject: BRANCH_A, + hackDir, + }); + releaseLock(); + await holder; + expect(locked.map((entry) => [entry.name, entry.result])).toEqual([ + [ + "recorded-helpers", + expect.objectContaining({ + state: "failed", + reason: "lock-unavailable", + }), + ], + ]); + }, + TIMEOUT +); + +test( + "a start that fails after a down keeps the shutdown in force for an earlier start", + async () => { + const root = await scratch(); + const helper = helperConfig({ port: freePort() }); + const hackDir = join(root, "hack-home"); + const earlier = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + // A down marks the instance stopping and releases its helpers. + await markSharedHelperInstanceStopping({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir, + }); + await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir, + }); + // A later start of the same instance fails before any helper is ready. + const failing = startProjectSharedHelpers({ + ...context(root), + helpers: [ + { ...helper, upstream: undefined, upstreamEnv: "UNSET_E2E_UPSTREAM" }, + ], + composeProject: BRANCH_A, + }); + await expect(failing).rejects.toThrow("UNSET_E2E_UPSTREAM"); + // The earlier start must still see the shutdown. + expect( + await earlier.assertCurrent().catch((caught: unknown) => caught) + ).toBeInstanceOf(SharedHelperSupersededError); + expect(await readHelperClaim(root, helper)).toBeNull(); + }, + TIMEOUT +); + +test( + "a start whose operation record cannot be written rolls back its holders", + async () => { + const root = await scratch(); + const helper = helperConfig({ port: freePort() }); + const instanceDir = join(root, "hack-home", "shared-helpers", ".instances"); + // A directory where the operation record belongs makes its write fail. + await mkdir(join(instanceDir, `${BRANCH_A}.operation`), { + recursive: true, + }); + await expect( + startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }) + ).rejects.toThrow(); + // The helper this start spawned was its only holder: it is stopped. + expect(await readHelperClaim(root, helper)).toBeNull(); + expect(await served(helper.port).catch(() => "")).toBe(""); + }, + TIMEOUT +); + +test( + "a start refuses to grow the acquired record past its limit; down still releases", + async () => { + const root = await scratch(); + const hackDir = join(root, "hack-home"); + const instanceDir = join(hackDir, "shared-helpers", ".instances"); + await mkdir(instanceDir, { recursive: true, mode: 0o700 }); + const record = join(instanceDir, `${BRANCH_A}.acquired.json`); + const full = Array.from({ length: 256 }, (_, index) => ({ + name: `old-helper-${index}`, + port: 20_000 + index, + })); + const fullText = `${JSON.stringify(full)}\n`; + await writeFile(record, fullText, { mode: 0o600 }); + const helper = helperConfig({ port: freePort() }); + await expect( + startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }) + ).rejects.toThrow("already records 256 shared helpers"); + expect(await readFile(record, "utf8")).toBe(fullText); + expect(await readHelperClaim(root, helper)).toBeNull(); + // The record stays readable, so down releases every entry and clears it. + const released = await releaseProjectSharedHelpers({ + helpers: undefined, + composeProject: BRANCH_A, + hackDir, + }); + expect(released).toHaveLength(256); + expect(released.every((entry) => entry.result.state === "absent")).toBe( + true + ); + expect(await readFile(record, "utf8").catch(() => null)).toBeNull(); + }, + TIMEOUT +); + +test( + "a later successful start does not restore an earlier start after a down, even with a disabled or changed helper", + async () => { + const root = await scratch(); + const hackDir = join(root, "hack-home"); + const helper = helperConfig({ port: freePort() }); + for (const later of [ + { ...helper, enabled: false }, + helperConfig({ port: freePort() }), + ]) { + const earlier = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + await markSharedHelperInstanceStopping({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir, + }); + await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir, + }); + const newer = await startProjectSharedHelpers({ + ...context(root), + helpers: [later], + composeProject: BRANCH_A, + }); + await newer.assertCurrent(); + expect( + await earlier.assertCurrent().catch((caught: unknown) => caught) + ).toBeInstanceOf(SharedHelperSupersededError); + await releaseProjectSharedHelpers({ + helpers: [later], + composeProject: BRANCH_A, + hackDir, + }); + } + }, + TIMEOUT +); + +function sessionOwnership(root: string, decisionTimeoutMs = 10_000) { + return createSharedHelperSessionOwnership({ + composeProject: BRANCH_A, + hackDir: join(root, "hack-home"), + decisionTimeoutMs, + }); +} + +test("a creator's session cleanup runs while the session is its own and is skipped once another start adopts it", async () => { + const root = await scratch(); + const cleaned: string[] = []; + const cleanup = (label: string) => () => { + cleaned.push(label); + return Promise.resolve(); + }; + const creator = sessionOwnership(root); + await creator.withLock(() => creator.record("created")); + expect(await creator.runIfOwned(cleanup("alone"))).toBe("ran"); + // Another start adopts the session: the creator's token would still kill it. + const adopter = sessionOwnership(root); + await adopter.withLock(() => adopter.record("adopted")); + expect(await creator.runIfOwned(cleanup("after-adoption"))).toBe("adopted"); + expect(cleaned).toEqual(["alone"]); +}); + +test("session ownership follows adoption, not the order starts ensured their helpers", async () => { + // Codex's sequence: A ensures helpers first, B ensures second and creates + // the session, A then adopts it and succeeds, and B fails in Compose. + const root = await scratch(); + const helper = helperConfig({ port: freePort() }); + await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + }); + const b = sessionOwnership(root); + const a = sessionOwnership(root); + await b.withLock(() => b.record("created")); + await a.withLock(() => a.record("adopted")); + let killed = false; + expect( + await b.runIfOwned(() => { + killed = true; + return Promise.resolve(); + }) + ).toBe("adopted"); + expect(killed).toBe(false); + await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir: join(root, "hack-home"), + }); +}); + +test("a creator's cleanup waits for an adoption decision in progress and then keeps the session", async () => { + const root = await scratch(); + const creator = sessionOwnership(root); + await creator.withLock(() => creator.record("created")); + const adopter = sessionOwnership(root); + let finishDecision = (): void => undefined; + const deciding = new Promise((resolveDecision) => { + finishDecision = resolveDecision; + }); + let inDecision = (): void => undefined; + const decisionStarted = new Promise((resolveStarted) => { + inDecision = resolveStarted; + }); + // The adopter has inspected the live session and is deciding to adopt it. + const adoption = adopter.withLock(async () => { + inDecision(); + await deciding; + await adopter.record("adopted"); + }); + await decisionStarted; + let killed = false; + const cleanup = creator.runIfOwned(() => { + killed = true; + return Promise.resolve(); + }); + await Bun.sleep(200); + expect(killed).toBe(false); + finishDecision(); + await adoption; + expect(await cleanup).toBe("adopted"); + expect(killed).toBe(false); +}); + +test( + "when the instance lock outlasts the cleanup timeout, a creator keeps the session another start may have adopted", + async () => { + const root = await scratch(); + const hackDir = join(root, "hack-home"); + const creator = sessionOwnership(root); + await creator.withLock(() => creator.record("created")); + // Another start adopted the session and succeeded. + const adopter = sessionOwnership(root); + await adopter.withLock(() => adopter.record("adopted")); + // A third start holds the instance lock (as a slow helper ensure would) + // for longer than the cleanup's lock timeout. + let releaseLock = (): void => undefined; + const held = new Promise((resolveHeld) => { + releaseLock = resolveHeld; + }); + let acquired = (): void => undefined; + const isHeld = new Promise((resolveAcquired) => { + acquired = resolveAcquired; + }); + const lockHolder = withProjectsRegistryLock({ + lockPath: join( + hackDir, + "shared-helpers", + ".instances", + `${BRANCH_A}.lock` + ), + run: async () => { + acquired(); + await held; + }, + }); + await isHeld; + let cleaned = false; + const outcome = await creator.runIfOwned(() => { + cleaned = true; + return Promise.resolve(); + }); + releaseLock(); + await lockHolder; + expect(outcome).toBe("undecidable"); + expect(cleaned).toBe(false); + }, + TIMEOUT +); + +test( + "a down during an operation's preparation hooks supersedes it before any helper starts", + async () => { + const root = await scratch(); + const hackDir = join(root, "hack-home"); + const helper = helperConfig({ port: freePort() }); + // Captured when the operation begins, before its up.before hooks. + const captured = await captureSharedHelperShutdownGeneration({ + composeProject: BRANCH_A, + hackDir, + }); + // A down completes while the hooks run. + await markSharedHelperInstanceStopping({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir, + }); + await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir, + }); + const error = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + shutdownGeneration: captured, + }).catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(SharedHelperSupersededError); + expect(await readHelperClaim(root, helper)).toBeNull(); + // A start that captures after the down proceeds normally. + const fresh = await startProjectSharedHelpers({ + ...context(root), + helpers: [helper], + composeProject: BRANCH_A, + shutdownGeneration: await captureSharedHelperShutdownGeneration({ + composeProject: BRANCH_A, + hackDir, + }), + }); + await fresh.assertCurrent(); + await releaseProjectSharedHelpers({ + helpers: [helper], + composeProject: BRANCH_A, + hackDir, + }); + }, + TIMEOUT +); diff --git a/tests/shared-helper-cleanup-combiner.test.ts b/tests/shared-helper-cleanup-combiner.test.ts new file mode 100644 index 000000000..1b75df230 --- /dev/null +++ b/tests/shared-helper-cleanup-combiner.test.ts @@ -0,0 +1,401 @@ +import { afterEach, expect, test } from "bun:test"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + assertSharedHelperSessionMuxSupported, + type LifecycleSharedHelpers, + releaseAfterFailedProcessStart, + retainHelpersUnlessContainersAbsent, + runGuardedSharedHelperStart, + signalCleanupKeepingStartedHelpers, + stopSessionBeforeReleasingHelpers, + withSharedHelperCleanup, +} from "../src/commands/project.ts"; +import { + LIFECYCLE_SESSION_RETAINED, + LifecycleSessionRetainedError, +} from "../src/lib/project-lifecycle-controller.ts"; +import { + listSharedHelpersHeld, + type ProcessCleanupOutcome, + resolveSharedHelperHolder, + withTerminationAbort, +} from "../src/lib/project-shared-helpers.ts"; +import type { MuxBackend } from "../src/mux/mux-backend.ts"; +import { setLoggerBackendOverride } from "../src/ui/logger.ts"; + +/** + * The combined lifecycle cleanup keeps the session and the helper holders + * together: when process cleanup is undecidable (the shared helper instance + * lock was busy), the helper rollback is skipped as well, so a kept session is + * never left without its helpers. + */ + +afterEach(() => { + setLoggerBackendOverride({ backend: null }); +}); + +function combined( + outcome: ProcessCleanupOutcome, + processResult: undefined | "session-retained" = undefined +) { + const events: string[] = []; + const sharedHelpers: LifecycleSharedHelpers = { + configured: true, + rollback: () => { + events.push("rollback"); + return Promise.resolve(); + }, + assertCurrent: () => Promise.resolve(), + }; + const sessionOwnership = { + runIfOwned: async (cleanup: () => Promise) => { + if (outcome === "ran") { + await cleanup(); + } + events.push(`processes:${outcome}`); + return outcome; + }, + }; + const result = withSharedHelperCleanup({ + cleanup: () => { + events.push("kill-session"); + return Promise.resolve(processResult); + }, + signalCleanup: null, + sharedHelpers, + sessionOwnership, + }); + return { events, cleanup: result.cleanup }; +} + +test("an undecidable process cleanup keeps the helper holders too", async () => { + const { events, cleanup } = combined("undecidable"); + await cleanup?.(); + expect(events).toEqual(["processes:undecidable"]); +}); + +test("a decided process cleanup is followed by the helper rollback", async () => { + for (const outcome of ["ran", "adopted"] as const) { + const { events, cleanup } = combined(outcome); + await cleanup?.(); + expect(events).toEqual([ + ...(outcome === "ran" ? ["kill-session"] : []), + `processes:${outcome}`, + "rollback", + ]); + } +}); + +test("a session kept because its stop was unconfirmed keeps the helper holders too", async () => { + const { events, cleanup } = combined("ran", "session-retained"); + await cleanup?.(); + expect(events).toEqual(["kill-session", "processes:ran"]); +}); + +test("a failed process start keeps the helper holders when its session was retained", async () => { + const cases = [ + { + name: "abort reports the retained session", + error: new Error("second process failed"), + aborted: LIFECYCLE_SESSION_RETAINED, + rolledBack: false, + }, + { + name: "the start already aborted and retained it", + error: new LifecycleSessionRetainedError(new Error("process failed")), + aborted: undefined, + rolledBack: false, + }, + { + name: "the session was stopped", + error: new Error("second process failed"), + aborted: undefined, + rolledBack: true, + }, + ] as const; + for (const entry of cases) { + let rolledBack = false; + await releaseAfterFailedProcessStart({ + error: entry.error, + abortProcesses: () => Promise.resolve(entry.aborted), + rollbackHelpers: () => { + rolledBack = true; + return Promise.resolve(); + }, + }); + expect({ name: entry.name, rolledBack }).toEqual({ + name: entry.name, + rolledBack: entry.rolledBack, + }); + } +}); + +test("down keeps every helper holder and reports them when the session stop failed or is unknown", async () => { + const released: string[] = []; + const outcome = await stopSessionBeforeReleasingHelpers({ + stopSession: () => + Promise.reject(new Error("Failed to stop owned lifecycle session")), + containers: "stopped", + releaseHelpers: () => { + released.push("release"); + return Promise.resolve([]); + }, + heldHelperNames: () => Promise.resolve(["chrome-bridge", "mail"]), + }); + expect(released).toEqual([]); + expect(outcome.sessionStopFailed?.error).toBeInstanceOf(Error); + // Callers (down, restart, native retirement) fail on a non-empty list. + expect(outcome.notReleased).toEqual(["chrome-bridge", "mail"]); +}); + +test("down releases the helper holders once the session stop succeeded", async () => { + const events: string[] = []; + const outcome = await stopSessionBeforeReleasingHelpers({ + stopSession: () => { + events.push("stop"); + return Promise.resolve(); + }, + containers: "stopped", + releaseHelpers: () => { + events.push("release"); + return Promise.resolve(["mail"]); + }, + heldHelperNames: () => Promise.resolve(["chrome-bridge", "mail"]), + }); + expect(events).toEqual(["stop", "release"]); + expect(outcome.sessionStopFailed).toBeNull(); + expect(outcome.notReleased).toEqual(["mail"]); +}); + +test("a signal during a startup whose session was retained never rolls the helpers back afterwards", async () => { + const events: string[] = []; + // Absorbs the SIGTERM withTerminationAbort re-raises once it settles. + const absorb = (): void => undefined; + process.on("SIGTERM", absorb); + let onSignalled: () => Promise = () => Promise.resolve(); + let signalled = false; + try { + const outcome = await withTerminationAbort( + () => + runGuardedSharedHelperStart({ + startHelpers: () => + Promise.resolve({ + configured: true, + rollback: () => { + events.push("rollback"); + return Promise.resolve(); + }, + assertCurrent: () => Promise.resolve(), + }), + startProcesses: async () => { + // SIGTERM arrives (and is caught) while processes start; the + // start then fails and its session stop cannot be confirmed. + process.kill(process.pid, "SIGTERM"); + await Bun.sleep(50); + throw new LifecycleSessionRetainedError(new Error("second failed")); + }, + abortProcesses: () => Promise.resolve(), + sessionOwnership: undefined, + setSignalCleanup: (cleanup) => { + onSignalled = cleanup; + }, + }), + { + onSignalled: () => { + signalled = true; + return onSignalled(); + }, + } + ).then( + () => "started", + (error: unknown) => error + ); + expect(outcome).toBeInstanceOf(LifecycleSessionRetainedError); + expect(signalled).toBe(true); + await Bun.sleep(20); + expect(events).toEqual([]); + } finally { + process.off("SIGTERM", absorb); + } +}); + +test("helpers recorded as acquired but removed from config are still reported as held", async () => { + const hackDir = await mkdtemp(join(tmpdir(), "hack-held-helpers-")); + try { + const composeProject = "demo--branch-a"; + const dir = join(hackDir, "shared-helpers", ".instances"); + await mkdir(dir, { recursive: true }); + const acquired = join( + dir, + `${resolveSharedHelperHolder(composeProject)}.acquired.json` + ); + // No helper is configured any more; the instance still holds `mail`. + expect( + await listSharedHelpersHeld({ helpers: [], composeProject, hackDir }) + ).toEqual([]); + await writeFile( + acquired, + `${JSON.stringify([{ name: "mail", port: 1025 }])}\n` + ); + expect( + await listSharedHelpersHeld({ helpers: [], composeProject, hackDir }) + ).toEqual(["mail"]); + // An unreadable record still reports something kept. + await writeFile(acquired, "not json"); + expect( + (await listSharedHelpersHeld({ helpers: [], composeProject, hackDir })) + .length + ).toBe(1); + } finally { + await rm(hackDir, { recursive: true, force: true }); + } +}); + +function resolvedMux(available: readonly ("tmux" | "zellij")[]) { + const backend = (name: "tmux" | "zellij") => + ({ + name, + available: available.includes(name), + createsWithOwnerToken: name === "tmux", + }) as unknown as MuxBackend; + return () => + Promise.resolve({ + mode: "auto" as const, + backends: new Map([ + ["tmux" as const, backend("tmux")], + ["zellij" as const, backend("zellij")], + ]), + }); +} + +test("shared helpers with a lifecycle session refuse zellij before anything starts", async () => { + await expect( + assertSharedHelperSessionMuxSupported({ + sharedHelpersConfigured: true, + needsSession: true, + mux: resolvedMux(["zellij"]), + }) + ).rejects.toThrow("need tmux"); + // tmux records the token with the session: allowed. + await assertSharedHelperSessionMuxSupported({ + sharedHelpersConfigured: true, + needsSession: true, + mux: resolvedMux(["tmux", "zellij"]), + }); + // No session, or no shared helpers: zellij is unaffected. + await assertSharedHelperSessionMuxSupported({ + sharedHelpersConfigured: true, + needsSession: false, + mux: resolvedMux(["zellij"]), + }); + await assertSharedHelperSessionMuxSupported({ + sharedHelpersConfigured: false, + needsSession: true, + mux: resolvedMux(["zellij"]), + }); +}); + +test("down keeps every helper holder when the containers were not confirmed stopped", async () => { + const events: string[] = []; + const outcome = await stopSessionBeforeReleasingHelpers({ + stopSession: () => { + events.push("stop"); + return Promise.resolve(); + }, + // Compose down failed or timed out (exit 124), or was interrupted. + containers: "unconfirmed", + releaseHelpers: () => { + events.push("release"); + return Promise.resolve([]); + }, + heldHelperNames: () => Promise.resolve(["mail"]), + }); + // The lifecycle session still stops; the helpers stay with the containers. + expect(events).toEqual(["stop"]); + expect(outcome.sessionStopFailed).toBeNull(); + expect(outcome.notReleased).toEqual(["mail"]); +}); + +test("containers kept on purpose (restart preserves the runtime) keep the holders without a failure", async () => { + const events: string[] = []; + const outcome = await stopSessionBeforeReleasingHelpers({ + stopSession: () => { + events.push("stop"); + return Promise.resolve(); + }, + containers: "kept", + releaseHelpers: () => { + events.push("release"); + return Promise.resolve([]); + }, + heldHelperNames: () => Promise.resolve(["mail"]), + }); + expect(events).toEqual(["stop"]); + expect(outcome).toEqual({ sessionStopFailed: null, notReleased: [] }); +}); + +test("once the containers run, the combined cleanup stops the processes but keeps the helpers", async () => { + const events: string[] = []; + const combinedCleanup = withSharedHelperCleanup({ + cleanup: () => { + events.push("processes"); + return Promise.resolve(undefined); + }, + signalCleanup: null, + sharedHelpers: { + configured: true, + rollback: () => { + events.push("rollback"); + return Promise.resolve(); + }, + assertCurrent: () => Promise.resolve(), + }, + sessionOwnership: undefined, + }); + // A detached Compose up succeeded; then an up.after hook fails. + combinedCleanup.keepSharedHelpers?.(); + await combinedCleanup.cleanup?.(); + expect(events).toEqual(["processes"]); +}); + +test("after an unsuccessful container start, helpers are kept unless Compose confirms no container may run", async () => { + const outcomes = [ + ["absent", false], + ["present", true], + ["unknown", true], + ["throws", true], + ] as const; + for (const [probe, kept] of outcomes) { + let keptHelpers = false; + await retainHelpersUnlessContainersAbsent({ + probe: () => + probe === "throws" + ? Promise.reject(new Error("docker unavailable")) + : Promise.resolve(probe), + keepSharedHelpers: () => { + keptHelpers = true; + }, + }); + expect([probe, keptHelpers]).toEqual([probe, kept]); + } +}); + +test("the signal cleanup keeps the helpers once a container start has begun", async () => { + for (const begun of [false, true]) { + const events: string[] = []; + const cleanup = signalCleanupKeepingStartedHelpers({ + cleanup: () => { + events.push("cleanup"); + return Promise.resolve(); + }, + containerStartBegun: () => begun, + keepSharedHelpers: () => () => { + events.push("keep"); + }, + }); + await cleanup?.(); + expect(events).toEqual(begun ? ["keep", "cleanup"] : ["cleanup"]); + } +}); diff --git a/tests/shared-host-lifecycle.test.ts b/tests/shared-host-lifecycle.test.ts index 1d9a00639..4d335daf6 100644 --- a/tests/shared-host-lifecycle.test.ts +++ b/tests/shared-host-lifecycle.test.ts @@ -382,6 +382,7 @@ test( state: "ready", ownership: "owned", spawned: true, + holderAdded: true, pid: owned.pid, endpoint: `http://127.0.0.1:${spec.port}`, }); @@ -406,6 +407,35 @@ test( TIMEOUT ); +test( + "holderAdded reports whether this call recorded the holder, so a rollback releases only its own", + async () => { + const root = await scratch(); + const spec = specFor({ root, port: freePort() }); + expect( + await ensureSharedHostHelper({ spec, holder: "branch-a" }) + ).toMatchObject({ state: "ready", spawned: true, holderAdded: true }); + const owned = await trackClaimedHelper(spec); + expect( + await ensureSharedHostHelper({ spec, holder: "branch-a" }) + ).toMatchObject({ state: "ready", spawned: false, holderAdded: false }); + expect( + await ensureSharedHostHelper({ spec, holder: "branch-b" }) + ).toMatchObject({ state: "ready", spawned: false, holderAdded: true }); + expect(await readClaim(spec)).toMatchObject({ + holders: ["branch-a", "branch-b"], + }); + expect(await releaseSharedHostHelper({ spec, holder: "branch-b" })).toEqual( + { state: "retained" } + ); + expect(await releaseSharedHostHelper({ spec, holder: "branch-a" })).toEqual( + { state: "stopped" } + ); + expect(await isAlive(owned)).toBe(false); + }, + TIMEOUT +); + test( "a healthy shared helper started elsewhere is adopted and never signalled on release", async () => { @@ -427,6 +457,7 @@ test( ownership: "adopted", endpoint: `http://127.0.0.1:${spec.port}`, spawned: false, + holderAdded: true, }); expect(seen.events).not.toContain("spawned"); expect(await (await fetch(`http://127.0.0.1:${spec.port}/`)).text()).toBe(