F1b: runner coordinator for attempts, slots and shutdown

[mchwang]

Lane F, step F1, slice F1b: runner coordinator. Related: #22, #51.

This replaces #56. GitHub closed #56 when #53 (F1a) merged and deleted #56's base branch, and a PR can't be reopened once its base is gone. This branch is #56 rebased onto main. The review history is on #56.

Changes since #56

• Rebased onto main. The unsquashed F1a commit and the contract round 35 commit were dropped, because F1a: persist the runner lifecycle in the Store #53's squash already contains both. The F1b commit applied without conflicts.
• New commit: pass D's runnerOwner through the coordinator. Lane D (D: label every resource with its runner, attempt and allocation (#51 item 3) #71) made InvocationInput.runnerOwner required, and D refuses task storage that belongs to another runner. RunnerDeps now has a runnerOwner, and the coordinator passes it to captureInvocation. The injected D prepares storage and starts invocations under that same token. F1d replaces the source with the per-database runner token.
• New commit: review fixes (26d9678).
  • Unreleased resources. When D settles with unreleased, the runner refuses all new work until restart, as Ask does. The resources are kept in memory and exposed as unreleased. They are not yet saved to the database: F1d adds that together with startup recovery.
  • No stranded handle. Before, if reading the attempt failed after a refused markRunning, the running D handle was dropped. Now the coordinator still cancels the handle and waits for it to settle.
  • No stranded job. Before, if reading the task budget failed right after admission, the job stayed in memory with no work and held its slot. Now the timers are set up inside the job, and a failed read ends the attempt as failed.
• New commits: review rounds 3–9 (7491e6b … db1d746).
  • Preparation files after launch. They are removed once D has settled and the terminal write succeeds. A failed terminal write leaves them for startup recovery.
  • Stale cause. stop(..., 'stale', cause) keeps the cause. The agent's stderr is never recorded as the reason an attempt went stale.
  • Preparation timeout. It ends failed "Timed out while preparing.", as the spec says. A later shutdown, time limit or stop can't replace it.
  • Launch check.
    • A context change is tested before the time checks, so it ends stale.
    • An attempt admitted before D reported unreleased is not started.
  • Fixed outcomes. Once a job is ending before launch, later stops are refused, so "stale, then shutdown" stays stale and isn't requeued. The same holds after the terminal write.
  • Cancel task and unsaved reasons.
    • An earlier stop reason that failed to save is saved before the Store writes cancelled. This runs only when the caller's state version is current.
    • A cancel the Store recorded on a job that no longer takes stops is shown in status() only.
    • The Store's rule stands: a pending cancel task wins.
  • Calls inside userAction. Retry and cancel are meant to run inside userAction.
    • A job starts its work only after the caller's transaction ends. If admission rolled back, it gives back the slot and prepares nothing.
    • A reason write that rolled back shows as unsaved.
    • A rolled-back cancel task never changes the outcome.
  • Fail-closed marker. It is set before the job is removed, so the slot is never free in between.
• New commit: Copilot review fixes (4b69ea3).
  • Only this attempt's result. A settled result whose attempt ID or captured context doesn't match is never validated or saved. The attempt fails closed, as the question path does.
  • Preparation files that can't be removed. If removing them fails, before launch or after the terminal write, the slot stays held under a preparation-not-removed marker until restart, and admission for the task says why.
  • Deferred to F1: apply a stop only after the caller's transaction commits #79. A stop inside a userAction transaction that rolls back still cancels D. Fixing it needs a Store post-commit hook.

What this does

Adds runner/coordinator.ts, the per-process owner of in-memory jobs, slots and unresolved markers, as specified in docs/implementation/runner-lifecycle.md. D is injected (RunnerDeps.start), so this runs against a fake D until #51 lands; nothing is wired to the server yet.

• Admission in two steps. In one synchronous turn, the coordinator checks it is open, that the task has no job or marker, and that a slot is free, then reserves the slot. Then Store.admitAttempt runs. A refused transaction releases the reservation in the same turn. Writable kinds share one slot and read-only kinds share another; both limits are configurable.
• Preparation. Host-side preparation gets an AbortSignal. The task budget timer records time-limit. The attempt deadline, before launch, fails the attempt with "Timed out" and records no first reason.
• Launch check. In one synchronous turn: the row is still pending, there is no first reason (saved or in memory), there is time left and the context is current. Then captureInvocation, then D's start. A context change during preparation ends stale without calling D. A stop during launch ends from its first reason. A start error ends failed with the launch error.
• Running. If markRunning is refused because a first reason exists, the handle is cancelled and settles normally. If the write fails with a storage error, the handle is cancelled with capture-failure, settled, and the slot stays held under a start-not-saved marker.
• Stops. The first reason wins. It is kept in memory and saved when possible; status().stopRequested.saved shows when saving failed. The slot is never freed on cancel, only after settled.
• Settlement. A clean result is validated, then the Store chooses the terminal state from the first reason, D's result and context currency. A failed terminal write leaves a result-not-saved marker that holds the slot until restart.
• Cancel task and shutdown. cancelTask goes through the Store and stops the running work. close() rejects admission, records shutdown only where no reason exists, and awaits every job with no timer. It leaves the Store open for the caller to close.

Validation (head 4b69ea3)

• npm run typecheck: passes.

• CI's unit set (npm test minus the Docker suites): 707 passed, 0 failed. The total rose from 488 because main gained tests. 45 of the tests are new, in test/runner-coordinator.test.ts, using a fake D whose promises the test controls, and real SQLite. The first test now also checks that D receives the injected runnerOwner. Each test added for a review fix fails without that fix (checked against the commit before it). The one exception is the case where saving the result and removing the files both fail, which guards existing behaviour. test/review.test.ts failed once in about ten runs; it doesn't touch the runner code.

• Mutation check (run on F1b: runner coordinator for attempts, slots and shutdown #56's head 3767065): six guards were broken one at a time, and each was caught:

  • a stop freeing the slot early;
  • the launch check skipping the context comparison;
  • no marker after a failed terminal write;
  • close() not waiting;
  • a later reason overwriting the first;
  • the reservation kept after a Store refusal.

• AGENTS.md race regressions covered here:

  • lease expiry or a clock jump, then a retry while the original still runs;
  • a timeout, then the provider stays unsettled, then a retry;
  • shutdown, then a new request;
  • an abort, then the subprocess settles later;
  • a context change before launch.

  The Store-level cases (an old attempt settling after a retry, two processes admitting at once) are in F1a.

• Browser tests weren't rerun: this slice doesn't touch web/.

Not in this slice

• F1c: the Store write gate, coordinator barriers and server wiring, /api/runner, and the merge-coordinator catches.
• F1d: startup recovery, the requeue claim and the OS lock.
• F1e: the planning endpoints, and feedback wiring into review actions.
• Task storage. Removing it after the terminal write is F2 (writable attempts).
• Recorded preparation process group. Saving preparation_pgid is part of F1d, together with the abortable D helpers from D follow-ups required by the F1 runner lifecycle contract #51.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

Transaction rollbacks can still cancel live work, and cleanup failures can release ownership prematurely.

Review effort: Balanced
Findings: 2 · 1

Open (3)

• Defer cancellation until transaction commit · New
• Reject stale responses from other attempts · New
• Preserve ownership when host cleanup fails · New

What changed in this PR

Adds the in-memory runner coordinator for attempt admission, execution, cancellation, settlement, slot ownership, and shutdown.

Changes:

• Coordinates writable and read-only attempt slots.
• Handles preparation, launch, stops, settlement, and unreleased resources.
• Adds comprehensive lifecycle and race regression tests.

File	Description
runner/​coordinator.ts	Implements runner lifecycle coordination.
test/​runner-coordinator.test.ts	Tests admission, races, failures, cancellation, and shutdown.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.