From 46db063082c807949dbad7a437cdc0a1c82a2bbd Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:30:38 -0500 Subject: [PATCH 01/48] docs(design): moving a working thread to another computer (Move v2 step 1) Plan for queueing a move until a thread is free, stopping it first, and letting an agent move its own thread, with the session fence and confirmed teardown that make it safe. Reviewed twice by the outside reviewer. --- docs/design/fleet.md | 179 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 179 insertions(+) diff --git a/docs/design/fleet.md b/docs/design/fleet.md index cea578694..cba808737 100644 --- a/docs/design/fleet.md +++ b/docs/design/fleet.md @@ -616,6 +616,185 @@ The two test servers: - Kill B mid-import. - A real Codex turn on the moved thread, to see the seed. +## Part 3b: Move a working thread (Nov 15, v2 step 1) + +Will's ask: keep a thread going on another computer when this one has to shut down. Move v1 refuses any thread with live work, so a working thread can't move. Step 1 adds a move that waits for the thread to be free, an option to stop the work first, and an agent tool. Clearing a whole computer from Fleet and the quit dialog is step 2, carrying the provider's own session is step 3, and families, goals, check-backs and PR automation are step 4. Step 1 changes none of v1's transfer protocol. A queued move ends in today's `thread.move.start`, and nothing about freezing, export, import or abort changes. + +### What the user sees + +- **Dialog.** When the only things stopping a move are live work, the dialog no longer refuses. It shows what the thread is doing and two choices: + - **Move when it's done** (default): the move starts as soon as the thread is free. + - **Stop and move now**: ends the thread's agent session, which stops the turn and all its background work, then moves. + + A thread that can't move for any other reason (room, children, goal, check-back, PR automation, General Chats) is still refused with today's words. + +- **While it waits.** A notice above the message box: "Moves to Windows PC when this turn ends." with Cancel. The text changes when it waits on something else: "...when its background work finishes", or "...once you answer" for an approval or question. While a move waits, the thread takes no new turns. The send button reads why. "Send when done" messages, steering and outside assistants' queued messages are refused with the same words. Approvals and questions still work, since the turn needs them to finish. +- **When it can no longer move.** If the turn leaves something that blocks a move (the agent set a check-back or a goal, started a child thread, turned on PR automation, or proposed a plan), the wait ends. The notice says why: "Didn't move: this thread's agent set itself a time to come back. Cancel it first, then move again." It stays until dismissed. The same happens when the thread is archived or deleted, the link is removed, or the target no longer has the chosen agent. +- **Target offline.** A move that is ready but can't reach the target keeps waiting and says "Waiting for Windows PC to connect." It starts when the link comes back. Cancel always works. +- **Sidebar.** The row shows the same "moving" state v1 shows once frozen, and a quieter "move queued" mark before that. + +### State + +`OrchestrationThread.pendingMove: { requestId, target: { environmentId, label }, agent: ModelSelection, mode: "when-done" | "stop-now", stage: "waiting" | "quiescing", requestedAt, requestedBy: { kind: "user" } | { kind: "agent", turnId } } | null`, projected into a new `pending_move_json` column (migration 111). A thread has at most one: a pending move or a move (`thread.move`), never both. + +Events: + +- `thread.move-queued` +- `thread.move-quiescing { requestId }` +- `thread.move-queue-ended { requestId, reason: "started" | "cancelled" | "blocked" | "gone", words? }` + +Commands: + +- `thread.move.queue`, from the user or from the agent tool; +- `thread.move.unqueue { requestId }`, Cancel; +- `thread.move.quiesce { requestId }`, internal, from the reactor; +- `thread.move.queue-end { requestId }`, internal, from the reactor. + +Every unqueue and queue-end names the request it ends, and the decider ignores one that names an older request. A late Cancel or an old blocked evaluation can't clear a newer queued move. + +### Who may queue (decider, one shared check) + +`threadMoveRefusal` becomes `threadMoveRefusals(model, thread) → Array<{ code, kind: "wait" | "blocked", words }>`. Every check runs, instead of the first one winning, so a busy thread doesn't hide a goal behind its running turn. + +- `wait`, gone on its own: + - a running turn, a waiting approval or question, a side turn; + - background tasks (both counts), a running command run or workflow run, voice active; + - another thread working in the same checkout. +- `blocked`, needs the user: + - everything else v1 refuses; + - **queued follow-up messages**: they would start the next turn as soon as this one ends ("Send or remove your queued messages first"); + - **a proposed plan waiting to be implemented**: `childHasLiveWork` counts it as live work, but only the user's answer clears it, and implementing it starts a turn ("Implement or dismiss the plan first"). + +`thread.move.queue` is refused when there is a `blocked` code, a pending move, or a move. It is allowed with zero or more `wait` codes. With no `wait` code it still queues, and the reactor starts it at once. That gives one path whether the thread is busy or not. + +For `mode: "stop-now"` the decision also plans: + +- `thread.session-stop-requested`: ends every runtime of the thread (ProviderCommandReactor `processSessionStopRequested`), and with them the running turn and the background work inside them; +- `thread.side-turn.interrupt` when a side turn runs (its runtime is separate); +- the existing stops for running command runs and workflow runs. + +Per-task stops and interrupts aren't used. Claude's interrupt deliberately leaves background tasks running (`perTaskStopAffordance`). Codex, ACP and Pi have no per-task stop, and Muse only has stop-all. An interrupt also projects `interrupted` before ACP and OpenCode finish cancelling. Ending the session is the one stop every driver completes. + +### The guard + +`threadMoveGuardRefusal` gains a pending-move rule. On a thread with `pendingMove`, it refuses one shared set of work-starting commands, `WORK_START_COMMANDS`: + +- `thread.turn.start`, `thread.turn.retry`, `thread.turn.resend`; +- `thread.follow-up.submit`, both steer and queue (scheduled runs into an existing thread use it too); +- `thread.side-turn.start`, `thread.realtime.start`, `thread.command-run.start`; +- `thread.goal.set`, `thread.participant.add`, `thread.pull-request-automation.set` turning a switch on, `thread.continue-at-reset.set` arming; +- `thread.create` with this thread as attached parent; +- implementing a plan. + +The automatic continue-at-reset arming inside `thread.session.set` (decider.ts:4143) is skipped while a move waits. The user asked to move, and that wins over an automatic retry here. Early agent start while typing is refused for the thread too. + +The words: "This thread is waiting to move to Windows PC. Cancel the move to keep working here." Everything a running turn needs keeps working: approvals, answers, activity, session updates, interrupt. The guard keeps the wait short and tells the user why. It is not what keeps the move safe. Providers start turns of their own, through runtime events rather than commands: Claude after queued steering or a finished background agent (ClaudeAdapter.ts:5150), ACP autonomous turns (AcpAdapter.ts:834), Codex `turn/started`. No command guard can refuse those. The barrier below handles them. + +### The barrier: the session ends before the freeze + +A thread is ready to move only when no runtime of it exists. The projected state can say "free" while a provider is still cancelling, or just before it starts a turn of its own. So the queue never goes straight from "free" to the freeze: + +1. When the projection has no `wait` code, the reactor dispatches `thread.move.quiesce { requestId }`. The decider records `thread.move-quiescing`, which sets `pendingMove.stage: "quiescing"`, and plans `thread.session-stop-requested`. In "stop-now" mode the queue decision records it straight away. From here the **session fence** holds (below). +2. It calls the new `ProviderService.quiesceThread(threadId)` and waits for it. This stops every session whose parsed thread id is this thread: main, participants, side runtimes (their own keys, ProviderCommandReactor.ts:5774) and early starts. Each driver confirms its teardown (below). The result is `confirmed`, or `unconfirmed { reason }`. + - `unconfirmed` keeps the move waiting, with "Couldn't confirm the agent stopped: ." plus Try again and Cancel. It never moves anyway. +3. It re-reads the projection. If a turn slipped in (a provider turn that ended before the stop, for example), it goes back to waiting. +4. Otherwise it calls `ThreadMoves.start`. `decideMoveStart` checks again, in the same decision as the freeze, that the session is stopped and there are no `wait` codes. If either fails, the move isn't started, and the reactor goes back to step 1. + +**The session fence.** A command guard can't stop sessions that reopen without a command. `serverReadSubagentTranscript` (ws.ts:1727) and context compaction both let ProviderService recover a stopped session (ProviderService.ts:792, 2205, 2793). So ProviderService itself asks one admission check before it starts or recovers any session for a thread: `ThreadSessionAdmission.check(threadId)`. The check reads the projection and refuses while `pendingMove.stage === "quiescing"` or `thread.move` is set, frozen or moved away. That covers starts, recovery, compaction, transcript reads that would recover, room agents, side runtimes and early starts. The fence lives in the projection, so it holds across restarts, and startup recovery skips fenced threads. v1 gains this fence too: today a frozen thread can be recovered by a transcript read. `thread.context-compact.request` joins `WORK_START_COMMANDS`. + +If the user cancels while the move is quiescing, the fence lifts with the pending move, and the next message starts a fresh session as usual. + +**Confirmed teardown, per driver.** `quiesceThread` passes only when each runtime confirms its work is gone. A session missing from `listSessions()` isn't enough: today Claude marks itself closed after a `query.close()` that may have failed (ClaudeAdapter.ts:8442-8482). ACP also ignores `session/close` failures, though Grok's background shells live in separate process groups that survive a plain kill (AcpSessionRuntime.ts:859-873). Each driver's stop gains a confirmed result: + +- **Claude:** wait for the CLI child process to exit after close, then kill its process group and wait for it. +- **Codex:** the app-server process exits. +- **ACP:** after `session/close`, or its timeout, kill the agent's process group and any process groups it reported for shells, then wait for exit. +- **OpenCode, Pi, Muse:** their host or process exits, after Muse's `task/stopAll`. + +A driver that can't prove it returns `unconfirmed`. These are the same checks v1's freeze should have made. v1's export also starts only after `quiesceThread` confirms. + +Once quiesced and fenced, nothing can start work. Provider turns need a session, and no session can open. A command the guard misses only makes the move wait. + +### The reactor (`moves/MoveQueue.ts`, beside ThreadMoves) + +It watches projected thread changes and link changes (added, removed, access changed, reconnected) for threads with `pendingMove`. It subscribes before its startup scan, so a change during the scan isn't missed, and the scan waits until provider session recovery has run. A move that is waiting on its target also retries on its own, every 15 s at first and backing off to 2 minutes, so a failed `peer.describe` on a connected link doesn't leave the thread stuck. For each one: + +1. A `blocked` code → `thread.move.queue-end { reason: "blocked", words }`. +2. A `wait` code → nothing. It looks again on the next change. +3. Free → the barrier above, then `ThreadMoves.start({ threadId, targetEnvironmentId, agent, queuedRequestId })`. `start` already re-runs preview and checks the target and the agent. + - A target refusal that can pass (unreachable, `peer.describe` failed) → keep waiting with "Waiting for to connect." + - A refusal that won't pass (link removed or off, agent gone, no project there) → `queue-end { reason: "blocked", words }`. + - Success → the decider's `decideMoveStart` records `thread.move-started` and, in the same decision, `thread.move-queue-ended { reason: "started" }` when the command carries the thread's current `queuedRequestId`. The pending move and the freeze swap in one commit. + +`decideMoveStart` refuses a `queuedRequestId` that isn't the thread's current one, so an old reactor pass can't start a cancelled move. A direct user `thread.move.start` on a thread with a pending move is refused ("A move is already waiting"). + +**Cancel against start.** `moves.unqueue { threadId, requestId }` dispatches `thread.move.unqueue`. If the move already started from that request (the thread's `move.queuedRequestId` matches), it becomes v1's `thread.move.cancel` for that transfer, which asks the target to drop it. The user sees one Cancel button either way. + +### Agent tool: `thread_move` + +- **Who gets it:** a room tool, main runtime only, in `THREAD_TOOL_NAMES`, access tier "changes". +- **What it moves:** the caller's own thread only, in step 1. +- **Input:** `{ computer, agent? }`. Computer names come from `thread_agents`' `computers`. `agent` defaults to the target's suggested agent, the way the dialog does. +- **What it does:** queues `mode: "when-done"` with `requestedBy: { kind: "agent", turnId }`. The caller's own turn is the live work, so the move starts once this turn ends, after the agent's last words. A turn that ends in an error or is interrupted still ends, and the move goes ahead. The user's Cancel is the way to keep the thread here. +- **What it returns:** "Queued: this thread moves to when your turn ends. Finish your reply; don't start more work." Or the refusal words. + +Fences: + +- refused on threads started by a link (`remoteOrigin`, already a `blocked` code) and on threads an outside agent works in (`outsideOrigin`); the endpoint's existing `changes` fence stays; +- refused for side runtimes; +- the target's own Ask first/Allowed/Off gate applies unchanged, as in v1. + +The user sees the same notice, with "Your agent asked to move this thread", and can Cancel. + +### Contracts + +- `ThreadMovePreview` gains `waitsFor?: ThreadMoveWait[]`: the wait codes with words. Only `blocked` codes fill `refusal`. +- New RPCs: `moves.queue { threadId, targetEnvironmentId, agent, mode }` and `moves.unqueue { threadId, requestId }`. The client sends the `requestId` it shows, never the server's current one. +- `moves.start` stays for older clients and for the reactor's path. + +### App guide + +The Fleet topic paragraph (added Nov 15) gains a sentence on Move when it's done, Stop and move now, and the agent tool. + +### Tests (focused) + +Decider: + +- queue refused on a `blocked` code; +- queue accepted while a turn runs; +- the guard refuses `thread.turn.start` and a "Send when done" follow-up while a move waits; +- a stale `queuedRequestId` is refused; +- start and queue-end happen in one decision; +- stop-now plans the session stop and the side-turn interrupt; +- the freeze is refused while the projected session isn't stopped; +- an unqueue or queue-end naming an old request changes nothing. + +Reactor, against a fake engine and links: + +- starts after the turn completes; +- not before background tasks settle; +- ends as blocked when the agent set a check-back during the turn; +- keeps waiting while the target is offline, then starts on reconnect; +- survives a restart with a pending move; +- a provider-started turn after "free" sends the move back to waiting; +- a Cancel that races the start cancels the transfer instead; +- a failed describe on a connected link retries. + +Browser: + +- the dialog's two choices on a working thread; +- the notice with Cancel; +- the send button's reason. + +One live check on the two-computer test stack with a fake fx agent: queue during a turn, the turn ends, the move commits, the thread continues on B. + +### Out of scope here (later steps) + +- clearing a computer from Fleet and the quit dialog (step 2); +- carrying native sessions (step 3); +- families, goals, check-backs, PR automation, rooms and General Chats (step 4); +- moving another thread with the agent tool. + ## Part 4: The rail and the Fleet page ### Rail From 35555f06d853321cacded3a887e084a0bc0aabf9 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:39:31 -0500 Subject: [PATCH 02/48] feat(contracts): a queued move that waits for a working thread to be free Move v2 step 1 (docs/design/fleet.md, Part 3b). A thread gains pendingMove, with its commands (thread.move.queue, .unqueue, .quiesce, .queue-status, .queue-end) and events (thread.move-queued, -quiescing, -queue-status-set, -queue-ended). thread.move.start and the projected move carry the queued request they started from, ThreadMovePreview gains waitsFor, and the moves RPCs (preview, start, cancel, plus the new queue, unqueue and retry) move into a group of their own. --- packages/contracts/src/baseSchemas.ts | 3 + packages/contracts/src/moves.ts | 49 ++++- packages/contracts/src/orchestration.ts | 236 ++++++++++++++++++++++++ packages/contracts/src/rpc.ts | 46 ++++- 4 files changed, 328 insertions(+), 6 deletions(-) diff --git a/packages/contracts/src/baseSchemas.ts b/packages/contracts/src/baseSchemas.ts index d415ba7c9..39d836a1b 100644 --- a/packages/contracts/src/baseSchemas.ts +++ b/packages/contracts/src/baseSchemas.ts @@ -83,6 +83,9 @@ export type ChildRequestId = typeof ChildRequestId.Type; /** One move of a thread to a linked computer (docs/design/fleet.md, Part 3). */ export const ThreadMoveId = makeEntityId("ThreadMoveId"); export type ThreadMoveId = typeof ThreadMoveId.Type; +/** One queued move of a thread, waiting for it to be free (docs/design/fleet.md, Part 3b). */ +export const ThreadMoveRequestId = makeEntityId("ThreadMoveRequestId"); +export type ThreadMoveRequestId = typeof ThreadMoveRequestId.Type; /** The requests one `thread_start` call made, approved or declined together. */ export const ChildRequestBatchId = makeEntityId("ChildRequestBatchId"); export type ChildRequestBatchId = typeof ChildRequestBatchId.Type; diff --git a/packages/contracts/src/moves.ts b/packages/contracts/src/moves.ts index d07c463f4..4c1a91eb7 100644 --- a/packages/contracts/src/moves.ts +++ b/packages/contracts/src/moves.ts @@ -5,10 +5,11 @@ import { NonNegativeInt, ThreadId, ThreadMoveId, + ThreadMoveRequestId, TrimmedNonEmptyString, } from "./baseSchemas.ts"; import { ComputerLinkAccess } from "./links.ts"; -import { ModelSelection } from "./orchestration.ts"; +import { ModelSelection, ThreadMoveWait, ThreadPendingMoveMode } from "./orchestration.ts"; import { PeerAgent } from "./peerRpc.ts"; /** @@ -48,8 +49,17 @@ export const ThreadMoveChanges = Schema.Struct({ export type ThreadMoveChanges = typeof ThreadMoveChanges.Type; export const ThreadMovePreview = Schema.Struct({ - /** Why this thread can't move at all right now. Absent: it can. */ + /** + * Why this thread can't move, even once it is free: something only the + * user can clear (a goal, queued messages, a room...). Absent: it can. + */ refusal: Schema.optional(TrimmedNonEmptyString), + /** + * What it is busy with now (docs/design/fleet.md, Part 3b). With no + * `refusal`, the dialog offers "Move when it's done" and "Stop and move + * now" (`moves.queue`). Absent or empty: it is free. + */ + waitsFor: Schema.optional(Schema.Array(ThreadMoveWait)), changes: Schema.optional(ThreadMoveChanges), targets: Schema.Array(ThreadMoveTargetPreview), }); @@ -72,6 +82,41 @@ export type ThreadMoveStartResult = typeof ThreadMoveStartResult.Type; export const ThreadMoveCancelInput = Schema.Struct({ threadId: ThreadId }); export type ThreadMoveCancelInput = typeof ThreadMoveCancelInput.Type; +/** + * Queue a move that starts once the thread is free (`when-done`), or once + * its agent session is ended (`stop-now`): docs/design/fleet.md, Part 3b. + * The thread's pending move (OrchestrationThread.pendingMove) shows it. + */ +export const ThreadMoveQueueInput = Schema.Struct({ + threadId: ThreadId, + targetEnvironmentId: EnvironmentId, + /** One of the target's agents, by its own instance id. */ + agent: ModelSelection, + mode: ThreadPendingMoveMode, +}); +export type ThreadMoveQueueInput = typeof ThreadMoveQueueInput.Type; + +export const ThreadMoveQueueResult = Schema.Struct({ requestId: ThreadMoveRequestId }); +export type ThreadMoveQueueResult = typeof ThreadMoveQueueResult.Type; + +/** + * Cancel a queued move: the `requestId` the client shows, never "whatever is + * queued now". If that move already started, its transfer is cancelled + * instead (as `moves.cancel` would). + */ +export const ThreadMoveUnqueueInput = Schema.Struct({ + threadId: ThreadId, + requestId: ThreadMoveRequestId, +}); +export type ThreadMoveUnqueueInput = typeof ThreadMoveUnqueueInput.Type; + +/** Try a held queued move again now (ThreadPendingMove.hold): its Try again. */ +export const ThreadMoveRetryInput = Schema.Struct({ + threadId: ThreadId, + requestId: ThreadMoveRequestId, +}); +export type ThreadMoveRetryInput = typeof ThreadMoveRetryInput.Type; + export class ThreadMoveError extends Schema.TaggedError()("ThreadMoveError", { detail: TrimmedNonEmptyString, }) { diff --git a/packages/contracts/src/orchestration.ts b/packages/contracts/src/orchestration.ts index d38632f4a..614ade03f 100644 --- a/packages/contracts/src/orchestration.ts +++ b/packages/contracts/src/orchestration.ts @@ -14,6 +14,7 @@ import { ChildRequestBatchId, ChildRequestId, ThreadMoveId, + ThreadMoveRequestId, CommandId, CommandRequestId, SettingsChangeRequestId, @@ -798,6 +799,12 @@ export const OrchestrationThreadMove = Schema.Struct({ /** The user asked to stop it; it stops once the other computer confirms. */ cancelRequested: Schema.Boolean, startedAt: IsoDateTime, + /** + * The queued move this one started from (ThreadPendingMove.requestId), so a + * Cancel the user pressed on the queued move cancels the transfer instead. + * Absent: started straight away. + */ + queuedRequestId: Schema.optional(ThreadMoveRequestId), /** The other computer took it: the thread there. */ movedAway: Schema.optional( Schema.Struct({ @@ -808,6 +815,110 @@ export const OrchestrationThreadMove = Schema.Struct({ }); export type OrchestrationThreadMove = typeof OrchestrationThreadMove.Type; +/** + * What a queued move waits for (docs/design/fleet.md, Part 3b). Each goes + * away on its own: the turn ends, the user answers, the background work + * finishes. `words` finish "Moves to …", e.g. "when this turn ends". + */ +export const ThreadMoveWaitCode = Schema.Literals([ + "turn", + "approval", + "question", + "side_turn", + "background", + "command_run", + "workflow_run", + "voice", + "shared_checkout", +]); +export type ThreadMoveWaitCode = typeof ThreadMoveWaitCode.Type; + +export const ThreadMoveWait = Schema.Struct({ + code: ThreadMoveWaitCode, + words: TrimmedNonEmptyString, +}); +export type ThreadMoveWait = typeof ThreadMoveWait.Type; + +/** + * `when-done`: the move starts once the thread is free. `stop-now`: the + * thread's agent session ends first (its turn and background work with it). + */ +export const ThreadPendingMoveMode = Schema.Literals(["when-done", "stop-now"]); +export type ThreadPendingMoveMode = typeof ThreadPendingMoveMode.Type; + +/** + * `waiting`: for the thread to be free. `quiescing`: free, and its agent + * sessions are being ended and proven gone; no session of the thread may + * start or recover from here (ThreadSessionAdmission on the server). + */ +export const ThreadPendingMoveStage = Schema.Literals(["waiting", "quiescing"]); +export type ThreadPendingMoveStage = typeof ThreadPendingMoveStage.Type; + +/** Who queued the move: the user, or the thread's own agent (`thread_move`) in that turn. */ +export const ThreadMoveRequester = Schema.Union([ + Schema.Struct({ kind: Schema.Literal("user") }), + Schema.Struct({ kind: Schema.Literal("agent"), turnId: TurnId }), +]); +export type ThreadMoveRequester = typeof ThreadMoveRequester.Type; + +/** + * Why a move that is ready doesn't start yet. `target_unreachable`: the + * other computer can't be reached ("Waiting for Mac mini to connect."). + * `stop_unconfirmed`: an agent's processes couldn't be proven gone ("Couldn't + * confirm the agent stopped: …"); the move never goes ahead without that. + */ +export const ThreadMoveHold = Schema.Struct({ + kind: Schema.Literals(["target_unreachable", "stop_unconfirmed"]), + words: TrimmedNonEmptyString, +}); +export type ThreadMoveHold = typeof ThreadMoveHold.Type; + +/** + * A move waiting for the thread to be free (docs/design/fleet.md, Part 3b). + * While one waits the thread takes no new work (approvals and answers the + * running turn needs still go through); once the thread is free the move + * becomes an ordinary move (OrchestrationThreadMove) in the same decision. + * A thread has a pending move or a move, never both. + */ +export const ThreadPendingMove = Schema.Struct({ + requestId: ThreadMoveRequestId, + target: ThreadMoveTarget, + /** The agent the other computer will run it with (its own instance id). */ + agent: ModelSelection, + mode: ThreadPendingMoveMode, + stage: ThreadPendingMoveStage, + requestedAt: IsoDateTime, + requestedBy: ThreadMoveRequester, + /** What it waits for now, as the server last saw it. Absent or empty: nothing. */ + waitsFor: Schema.optional(Schema.Array(ThreadMoveWait)), + /** Why a move that is ready doesn't start yet. Absent or null: nothing holds it. */ + hold: Schema.optional(Schema.NullOr(ThreadMoveHold)), +}); +export type ThreadPendingMove = typeof ThreadPendingMove.Type; + +/** How a queued move ended: it started, the user cancelled it, it can't go, or the thread is gone. */ +export const ThreadPendingMoveEndReason = Schema.Literals([ + "started", + "cancelled", + "blocked", + "gone", +]); +export type ThreadPendingMoveEndReason = typeof ThreadPendingMoveEndReason.Type; + +/** + * Activity appended when a queued move ends without starting (blocked, or + * the thread archived or deleted): "Didn't move: ". Its payload is a + * ThreadMoveNotStartedPayload. + */ +export const ThreadMoveNotStartedActivityKind = "thread.move-not-started"; +export const ThreadMoveNotStartedPayload = Schema.Struct({ + requestId: ThreadMoveRequestId, + targetLabel: TrimmedNonEmptyString, + reason: Schema.Literals(["blocked", "gone"]), + words: TrimmedNonEmptyString, +}); +export type ThreadMoveNotStartedPayload = typeof ThreadMoveNotStartedPayload.Type; + /** A thread that moved here from another computer, and where it came from. */ export const ThreadMovedFrom = Schema.Struct({ environmentId: EnvironmentId, @@ -2537,6 +2648,8 @@ export const OrchestrationThread = Schema.Struct({ outsideWork: Schema.optional(Schema.Array(OrchestrationOutsideWork)), /** See OrchestrationThreadShell.move. */ move: Schema.optional(Schema.NullOr(OrchestrationThreadMove)), + /** See OrchestrationThreadShell.pendingMove. */ + pendingMove: Schema.optional(Schema.NullOr(ThreadPendingMove)), /** See OrchestrationThreadShell.movedFrom. */ movedFrom: Schema.optional(Schema.NullOr(ThreadMovedFrom)), /** @@ -2761,6 +2874,8 @@ export const OrchestrationThreadShell = Schema.Struct({ remoteOrigin: Schema.optional(Schema.NullOr(ThreadRemoteOrigin)), /** Moving, or moved, to a linked computer: see OrchestrationThreadMove. Absent or null: neither. */ move: Schema.optional(Schema.NullOr(OrchestrationThreadMove)), + /** A move waiting for the thread to be free: see ThreadPendingMove. Absent or null: none. */ + pendingMove: Schema.optional(Schema.NullOr(ThreadPendingMove)), /** Moved here from another computer: see ThreadMovedFrom. Absent or null: started here. */ movedFrom: Schema.optional(Schema.NullOr(ThreadMovedFrom)), /** @@ -4674,6 +4789,68 @@ const ThreadMoveStartCommand = Schema.Struct({ transferId: ThreadMoveId, target: ThreadMoveTarget, agent: ModelSelection, + /** + * The thread's queued move this starts (the move queue's). It must be the + * thread's current one; the queued move ends in the same decision. + */ + queuedRequestId: Schema.optional(ThreadMoveRequestId), + createdAt: IsoDateTime, +}); + +/** + * A move that waits for the thread to be free (docs/design/fleet.md, Part + * 3b), on the computer it leaves. Only the move queue dispatches these, after + * it checked the link (`moves.queue`, or the agent's `thread_move`). + */ +const ThreadMoveQueueCommand = Schema.Struct({ + type: Schema.Literal("thread.move.queue"), + commandId: CommandId, + threadId: ThreadId, + requestId: ThreadMoveRequestId, + target: ThreadMoveTarget, + agent: ModelSelection, + mode: ThreadPendingMoveMode, + requestedBy: ThreadMoveRequester, + createdAt: IsoDateTime, +}); + +/** Cancel: ends the queued move `requestId`, and never a newer one. */ +const ThreadMoveUnqueueCommand = Schema.Struct({ + type: Schema.Literal("thread.move.unqueue"), + commandId: CommandId, + threadId: ThreadId, + requestId: ThreadMoveRequestId, + createdAt: IsoDateTime, +}); + +/** The thread is free: its sessions end, and none may start until the move starts or ends. */ +const ThreadMoveQuiesceCommand = Schema.Struct({ + type: Schema.Literal("thread.move.quiesce"), + commandId: CommandId, + threadId: ThreadId, + requestId: ThreadMoveRequestId, + createdAt: IsoDateTime, +}); + +/** What the queued move waits for now, and what holds it once it is ready. */ +const ThreadMoveQueueStatusCommand = Schema.Struct({ + type: Schema.Literal("thread.move.queue-status"), + commandId: CommandId, + threadId: ThreadId, + requestId: ThreadMoveRequestId, + waitsFor: Schema.Array(ThreadMoveWait), + hold: Schema.NullOr(ThreadMoveHold), + createdAt: IsoDateTime, +}); + +/** The queued move can't go (`blocked`, with why), or its thread is archived or deleted (`gone`). */ +const ThreadMoveQueueEndCommand = Schema.Struct({ + type: Schema.Literal("thread.move.queue-end"), + commandId: CommandId, + threadId: ThreadId, + requestId: ThreadMoveRequestId, + reason: Schema.Literals(["blocked", "gone"]), + words: TrimmedNonEmptyString, createdAt: IsoDateTime, }); @@ -4823,6 +5000,11 @@ const InternalOrchestrationCommand = Schema.Union([ ThreadTurnResendCommand, ProjectGeneralChatsRelocateCommand, ThreadMoveStartCommand, + ThreadMoveQueueCommand, + ThreadMoveUnqueueCommand, + ThreadMoveQuiesceCommand, + ThreadMoveQueueStatusCommand, + ThreadMoveQueueEndCommand, ThreadMoveProgressCommand, ThreadMoveCancelCommand, ThreadMoveAbortCommand, @@ -4930,6 +5112,10 @@ export const OrchestrationEventType = Schema.Literals([ "thread.remote-cancel-queued", "thread.remote-cancels-delivered", "thread.move-started", + "thread.move-queued", + "thread.move-quiescing", + "thread.move-queue-status-set", + "thread.move-queue-ended", "thread.move-progressed", "thread.move-cancel-requested", "thread.move-aborted", @@ -5270,6 +5456,36 @@ export const ThreadMoveStartedPayload = Schema.Struct({ target: ThreadMoveTarget, agent: ModelSelection, startedAt: IsoDateTime, + /** See OrchestrationThreadMove.queuedRequestId. */ + queuedRequestId: Schema.optional(ThreadMoveRequestId), +}); + +export const ThreadMoveQueuedPayload = Schema.Struct({ + threadId: ThreadId, + pendingMove: ThreadPendingMove, +}); + +export const ThreadMoveQuiescingPayload = Schema.Struct({ + threadId: ThreadId, + requestId: ThreadMoveRequestId, + updatedAt: IsoDateTime, +}); + +export const ThreadMoveQueueStatusSetPayload = Schema.Struct({ + threadId: ThreadId, + requestId: ThreadMoveRequestId, + waitsFor: Schema.Array(ThreadMoveWait), + hold: Schema.NullOr(ThreadMoveHold), + updatedAt: IsoDateTime, +}); + +export const ThreadMoveQueueEndedPayload = Schema.Struct({ + threadId: ThreadId, + requestId: ThreadMoveRequestId, + reason: ThreadPendingMoveEndReason, + /** Why, for `blocked` and `gone`. */ + words: Schema.optional(TrimmedNonEmptyString), + endedAt: IsoDateTime, }); export const ThreadMoveProgressedPayload = Schema.Struct({ @@ -6008,6 +6224,26 @@ export const OrchestrationEvent = Schema.Union([ type: Schema.Literal("thread.move-started"), payload: ThreadMoveStartedPayload, }), + Schema.Struct({ + ...EventBaseFields, + type: Schema.Literal("thread.move-queued"), + payload: ThreadMoveQueuedPayload, + }), + Schema.Struct({ + ...EventBaseFields, + type: Schema.Literal("thread.move-quiescing"), + payload: ThreadMoveQuiescingPayload, + }), + Schema.Struct({ + ...EventBaseFields, + type: Schema.Literal("thread.move-queue-status-set"), + payload: ThreadMoveQueueStatusSetPayload, + }), + Schema.Struct({ + ...EventBaseFields, + type: Schema.Literal("thread.move-queue-ended"), + payload: ThreadMoveQueueEndedPayload, + }), Schema.Struct({ ...EventBaseFields, type: Schema.Literal("thread.move-progressed"), diff --git a/packages/contracts/src/rpc.ts b/packages/contracts/src/rpc.ts index 3506c9457..95a5653fe 100644 --- a/packages/contracts/src/rpc.ts +++ b/packages/contracts/src/rpc.ts @@ -106,8 +106,12 @@ import { ThreadMoveError, ThreadMovePreview, ThreadMovePreviewInput, + ThreadMoveQueueInput, + ThreadMoveQueueResult, + ThreadMoveRetryInput, ThreadMoveStartInput, ThreadMoveStartResult, + ThreadMoveUnqueueInput, } from "./moves.ts"; import { ProjectSetupDestination, @@ -779,6 +783,9 @@ export const WS_METHODS = { movesPreview: "moves.preview", movesStart: "moves.start", movesCancel: "moves.cancel", + movesQueue: "moves.queue", + movesUnqueue: "moves.unqueue", + movesRetry: "moves.retry", // Setting up a project on another computer (docs/design/fleet.md, Part 8) projectSetupsSource: "projectSetups.source", @@ -2311,6 +2318,39 @@ export const WsMovesCancelRpc = Rpc.make(WS_METHODS.movesCancel, { error: ThreadMoveError, }); +/** Queue a move that starts once the thread is free, or once its agent is stopped. */ +export const WsMovesQueueRpc = Rpc.make(WS_METHODS.movesQueue, { + payload: ThreadMoveQueueInput, + success: ThreadMoveQueueResult, + error: ThreadMoveError, +}); + +/** Cancel a queued move, or the transfer it already started. */ +export const WsMovesUnqueueRpc = Rpc.make(WS_METHODS.movesUnqueue, { + payload: ThreadMoveUnqueueInput, + error: ThreadMoveError, +}); + +/** Try a held queued move again now. */ +export const WsMovesRetryRpc = Rpc.make(WS_METHODS.movesRetry, { + payload: ThreadMoveRetryInput, + error: ThreadMoveError, +}); + +/** + * Moving threads to linked computers. A group of its own, merged into + * WsRpcGroup, for the same reason as the outside agents' group: the main + * group sits near TypeScript's cap on type work per statement. + */ +export const WsMovesRpcGroup = RpcGroup.make( + WsMovesPreviewRpc, + WsMovesStartRpc, + WsMovesCancelRpc, + WsMovesQueueRpc, + WsMovesUnqueueRpc, + WsMovesRetryRpc, +); + /** On the source computer: what another computer needs to copy this project. */ export const WsProjectSetupsSourceRpc = Rpc.make(WS_METHODS.projectSetupsSource, { payload: ProjectSetupSourceInput, @@ -2697,7 +2737,7 @@ export const WsThemeShowcaseRpcGroup = RpcGroup.make( WsThemeShowcaseTakeBackRpc, ); -/** Everything else the WebSocket serves; with the outside agents', themes' and Showcase's groups, WsRpcGroup. */ +/** Everything else the WebSocket serves; with the outside agents', themes', Showcase's and moves' groups, WsRpcGroup. */ export const WsCoreRpcGroup = RpcGroup.make( WsServerGetConfigRpc, WsServerRefreshProvidersRpc, @@ -2903,9 +2943,6 @@ export const WsCoreRpcGroup = RpcGroup.make( WsScheduledTasksTestWebhookRpc, WsScheduledTasksRunWithWebhookEventRpc, WsScheduledTasksDismissWebhookAddressChangeRpc, - WsMovesPreviewRpc, - WsMovesStartRpc, - WsMovesCancelRpc, WsProjectSetupsSourceRpc, WsProjectSetupsReadFilesRpc, WsProjectSetupsInspectDestinationRpc, @@ -2933,4 +2970,5 @@ export const WsRpcGroup = WsCoreRpcGroup.merge( WsOutsideAgentsRpcGroup, WsThemesRpcGroup, WsThemeShowcaseRpcGroup, + WsMovesRpcGroup, ); From d0fab6d2017eef2140bd7627a1fd646989e0fe6b Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:53:04 -0500 Subject: [PATCH 03/48] feat(server): the decider queues a move until its thread is free threadMoveRefusals runs every check and sorts each into a wait (a running turn, an approval, background work, another thread busy in the same folder) or a block only the user clears (queued messages, a plan to act on, plus everything v1 refuses). thread.move.queue is refused on a block and allowed with any waits; while a move waits, the guard refuses the commands that start work, and the automatic continue-at-reset arming is skipped. thread.move.quiesce, .queue-status, .queue-end and .unqueue each name their request, so an old one changes nothing. A start from the queue must name the current request and find the session stopped; the pending move ends in the same decision. The projection keeps pendingMove in pending_move_json (migration 111). --- .../Layers/CheckpointReactor.test.ts | 1 + .../Layers/ProjectionPipeline.ts | 25 + .../Layers/ProjectionSnapshotQuery.ts | 11 +- .../Layers/ProviderCommandReactor.test.ts | 1 + .../Layers/ProviderRuntimeIngestion.test.ts | 1 + .../orchestration/decider.threadMoves.test.ts | 233 +++++++ apps/server/src/orchestration/decider.ts | 50 ++ .../orchestration/pendingMoveProjection.ts | 35 + apps/server/src/orchestration/projector.ts | 19 + .../src/orchestration/threadMoveDecisions.ts | 618 ++++++++++++++++-- .../persistence/Layers/ProjectionThreads.ts | 7 + apps/server/src/persistence/Migrations.ts | 2 + .../Migrations/111_ThreadPendingMove.ts | 14 + .../persistence/Services/ProjectionThreads.ts | 3 + 14 files changed, 954 insertions(+), 66 deletions(-) create mode 100644 apps/server/src/orchestration/pendingMoveProjection.ts create mode 100644 apps/server/src/persistence/Migrations/111_ThreadPendingMove.ts diff --git a/apps/server/src/orchestration/Layers/CheckpointReactor.test.ts b/apps/server/src/orchestration/Layers/CheckpointReactor.test.ts index 8886f517a..5f1ba0cd1 100644 --- a/apps/server/src/orchestration/Layers/CheckpointReactor.test.ts +++ b/apps/server/src/orchestration/Layers/CheckpointReactor.test.ts @@ -154,6 +154,7 @@ function createProviderServiceHarness( respondToUserInput: () => unsupported(), stopSession: () => unsupported(), listSessions, + quiesceThread: () => Effect.succeed({ confirmed: true } as const), getCapabilities: () => Effect.succeed({ sessionModelSwitch: "in-session" }), getInstanceInfo: (instanceId) => Effect.succeed({ diff --git a/apps/server/src/orchestration/Layers/ProjectionPipeline.ts b/apps/server/src/orchestration/Layers/ProjectionPipeline.ts index 46a2d7237..c4f4dece3 100644 --- a/apps/server/src/orchestration/Layers/ProjectionPipeline.ts +++ b/apps/server/src/orchestration/Layers/ProjectionPipeline.ts @@ -42,6 +42,7 @@ import { } from "@threadlines/shared/pendingRequests"; import { toPersistenceSqlError, type ProjectionRepositoryError } from "../../persistence/Errors.ts"; +import { nextPendingMove } from "../pendingMoveProjection.ts"; import { OrchestrationEventStore } from "../../persistence/Services/OrchestrationEventStore.ts"; import { ProjectionPendingApprovalRepository } from "../../persistence/Services/ProjectionPendingApprovals.ts"; import { ProjectionProjectRepository } from "../../persistence/Services/ProjectionProjects.ts"; @@ -561,6 +562,7 @@ const makeOrchestrationProjectionPipeline = Effect.fn("makeOrchestrationProjecti worktreeOrigin: event.payload.worktreeOrigin ?? null, outsideOrigin: event.payload.outsideOrigin ?? null, move: null, + pendingMove: null, movedFrom: event.payload.movedFrom ?? null, historySeedOwed: event.payload.historySeedOwed === true ? 1 : 0, effectiveCwd: null, @@ -646,6 +648,9 @@ const makeOrchestrationProjectionPipeline = Effect.fn("makeOrchestrationProjecti phase: "exporting" as const, cancelRequested: false, startedAt: event.payload.startedAt, + ...(event.payload.queuedRequestId !== undefined + ? { queuedRequestId: event.payload.queuedRequestId } + : {}), } : move === null || move.transferId !== event.payload.transferId ? move @@ -678,6 +683,26 @@ const makeOrchestrationProjectionPipeline = Effect.fn("makeOrchestrationProjecti return; } + // A move waiting for the thread to be free: the row keeps it as + // ThreadPendingMove, by the same rule as the in-memory projector. + case "thread.move-queued": + case "thread.move-quiescing": + case "thread.move-queue-status-set": + case "thread.move-queue-ended": { + const existingRow = yield* projectionThreadRepository.getById({ + threadId: event.payload.threadId, + }); + if (Option.isNone(existingRow)) return; + const pendingMove = existingRow.value.pendingMove ?? null; + const nextPending = nextPendingMove(pendingMove, event); + if (nextPending === pendingMove) return; + yield* projectionThreadRepository.upsert({ + ...existingRow.value, + pendingMove: nextPending, + }); + return; + } + case "thread.history-seed-delivered": { const existingRow = yield* projectionThreadRepository.getById({ threadId: event.payload.threadId, diff --git a/apps/server/src/orchestration/Layers/ProjectionSnapshotQuery.ts b/apps/server/src/orchestration/Layers/ProjectionSnapshotQuery.ts index c4092875a..5f9dadecd 100644 --- a/apps/server/src/orchestration/Layers/ProjectionSnapshotQuery.ts +++ b/apps/server/src/orchestration/Layers/ProjectionSnapshotQuery.ts @@ -63,6 +63,7 @@ import { RoomAgentInvite, ThreadMessageOrigin, OrchestrationThreadMove, + ThreadPendingMove, ThreadMovedFrom, ThreadRemoteOrigin, ThreadScheduledRun, @@ -173,6 +174,7 @@ type ThreadLineageRow = Pick< | "outsideOrigin" | "outsideWork" | "move" + | "pendingMove" | "movedFrom" | "historySeedOwed" | "historySeedMessageId" @@ -199,8 +201,9 @@ const threadLineageFields = (row: ThreadLineageRow) => ({ }); /** Moving a thread to a linked computer, or moved here from one (OrchestrationThreadMove). */ -const threadMoveFields = (row: Pick) => ({ +const threadMoveFields = (row: Pick) => ({ ...(row.move ? { move: row.move } : {}), + ...(row.pendingMove ? { pendingMove: row.pendingMove } : {}), ...(row.movedFrom ? { movedFrom: row.movedFrom } : {}), }); @@ -302,6 +305,7 @@ const ProjectionThreadDbRowSchema = ProjectionThread.mapFields( Schema.NullOr(Schema.fromJsonString(Schema.Array(OrchestrationOutsideWork))), ), move: Schema.optional(Schema.NullOr(Schema.fromJsonString(OrchestrationThreadMove))), + pendingMove: Schema.optional(Schema.NullOr(Schema.fromJsonString(ThreadPendingMove))), movedFrom: Schema.optional(Schema.NullOr(Schema.fromJsonString(ThreadMovedFrom))), }), ); @@ -945,6 +949,7 @@ const makeProjectionSnapshotQuery = Effect.gen(function* () { outside_origin_json AS "outsideOrigin", outside_work_json AS "outsideWork", move_json AS "move", + pending_move_json AS "pendingMove", moved_from_json AS "movedFrom", history_seed_owed AS "historySeedOwed", history_seed_message_id AS "historySeedMessageId", @@ -1013,6 +1018,7 @@ const makeProjectionSnapshotQuery = Effect.gen(function* () { outside_origin_json AS "outsideOrigin", outside_work_json AS "outsideWork", move_json AS "move", + pending_move_json AS "pendingMove", moved_from_json AS "movedFrom", history_seed_owed AS "historySeedOwed", history_seed_message_id AS "historySeedMessageId", @@ -1083,6 +1089,7 @@ const makeProjectionSnapshotQuery = Effect.gen(function* () { outside_origin_json AS "outsideOrigin", outside_work_json AS "outsideWork", move_json AS "move", + pending_move_json AS "pendingMove", moved_from_json AS "movedFrom", history_seed_owed AS "historySeedOwed", history_seed_message_id AS "historySeedMessageId", @@ -1916,6 +1923,7 @@ const makeProjectionSnapshotQuery = Effect.gen(function* () { outside_origin_json AS "outsideOrigin", outside_work_json AS "outsideWork", move_json AS "move", + pending_move_json AS "pendingMove", moved_from_json AS "movedFrom", history_seed_owed AS "historySeedOwed", history_seed_message_id AS "historySeedMessageId", @@ -3712,6 +3720,7 @@ const makeProjectionSnapshotQuery = Effect.gen(function* () { outside_origin_json AS "outsideOrigin", outside_work_json AS "outsideWork", move_json AS "move", + pending_move_json AS "pendingMove", moved_from_json AS "movedFrom", history_seed_owed AS "historySeedOwed", history_seed_message_id AS "historySeedMessageId", diff --git a/apps/server/src/orchestration/Layers/ProviderCommandReactor.test.ts b/apps/server/src/orchestration/Layers/ProviderCommandReactor.test.ts index 94c9cf702..f4eb57123 100644 --- a/apps/server/src/orchestration/Layers/ProviderCommandReactor.test.ts +++ b/apps/server/src/orchestration/Layers/ProviderCommandReactor.test.ts @@ -545,6 +545,7 @@ describe("ProviderCommandReactor", () => { respondToUserInput: respondToUserInput as ProviderServiceShape["respondToUserInput"], stopSession: stopSession as ProviderServiceShape["stopSession"], listSessions: () => Effect.succeed(runtimeSessions), + quiesceThread: () => Effect.succeed({ confirmed: true } as const), getCapabilities: (_provider) => Effect.succeed({ sessionModelSwitch: input?.sessionModelSwitch ?? "in-session", diff --git a/apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.test.ts b/apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.test.ts index 16ee4d302..e3681e9b0 100644 --- a/apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.test.ts +++ b/apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.test.ts @@ -145,6 +145,7 @@ function createProviderServiceHarness() { respondToUserInput: () => unsupported(), stopSession: () => unsupported(), listSessions: () => Effect.succeed([...runtimeSessions]), + quiesceThread: () => Effect.succeed({ confirmed: true } as const), getCapabilities: () => Effect.succeed({ sessionModelSwitch: "in-session" }), getInstanceInfo: (instanceId) => { const driverKind = ProviderDriverKind.make(String(instanceId)); diff --git a/apps/server/src/orchestration/decider.threadMoves.test.ts b/apps/server/src/orchestration/decider.threadMoves.test.ts index 8eed8d5a4..16639dac5 100644 --- a/apps/server/src/orchestration/decider.threadMoves.test.ts +++ b/apps/server/src/orchestration/decider.threadMoves.test.ts @@ -5,19 +5,24 @@ import { MessageId, ProjectId, ProviderInstanceId, + SideTurnId, ThreadId, ThreadMoveId, + ThreadMoveRequestId, TurnId, type ModelSelection, type OrchestrationCommand, type OrchestrationEvent, type OrchestrationReadModel, + type OrchestrationSession, + type ThreadPendingMoveMode, } from "@threadlines/contracts"; import * as Effect from "effect/Effect"; import { describe, expect, it } from "vite-plus/test"; import { decideOrchestrationCommand } from "./decider.ts"; import { createEmptyReadModel, projectEvent } from "./projector.ts"; +import { threadMoveGuardRefusal } from "./threadMoveDecisions.ts"; const now = "2026-10-22T10:00:00.000Z"; const modelSelection: ModelSelection = { @@ -357,3 +362,231 @@ describe("moving a thread to a linked computer", () => { expect(threadOf(kept, ARRIVED).historySeedOwed).toBe(false); }); }); + +describe("a move that waits for the thread to be free", () => { + const REQUEST = ThreadMoveRequestId.make("request-1"); + const queue = ( + mode: ThreadPendingMoveMode = "when-done", + requestId = REQUEST, + ): OrchestrationCommand => ({ + type: "thread.move.queue", + commandId: nextCommandId(), + threadId: THREAD, + requestId, + target: MAC_MINI, + agent: modelSelection, + mode, + requestedBy: { kind: "user" }, + createdAt: now, + }); + const startQueued = (queuedRequestId = REQUEST): OrchestrationCommand => ({ + type: "thread.move.start", + commandId: nextCommandId(), + threadId: THREAD, + transferId: TRANSFER, + target: MAC_MINI, + agent: modelSelection, + queuedRequestId, + createdAt: now, + }); + const quiesce = (requestId = REQUEST): OrchestrationCommand => ({ + type: "thread.move.quiesce", + commandId: nextCommandId(), + threadId: THREAD, + requestId, + createdAt: now, + }); + const withSession = ( + model: OrchestrationReadModel, + status: OrchestrationSession["status"], + activeTurnId: TurnId | null = null, + ) => + project(model, "thread.session-set", { + threadId: THREAD, + session: { + threadId: THREAD, + status, + providerName: "codex", + runtimeMode: "auto", + activeTurnId, + lastError: null, + updatedAt: now, + }, + }); + const decided = (model: OrchestrationReadModel, command: OrchestrationCommand) => + Effect.runPromise(decideOrchestrationCommand({ command, readModel: model })).then((events) => + (Array.isArray(events) ? events : [events]).map((event) => event.type), + ); + const turnStart = (): OrchestrationCommand => + ({ + type: "thread.turn.start", + commandId: nextCommandId(), + threadId: THREAD, + message: { messageId: MessageId.make("m-new"), role: "user", text: "more", attachments: [] }, + runtimeMode: "auto", + interactionMode: "default", + createdAt: now, + }) as OrchestrationCommand; + + it("queues while a turn runs, refuses new work meanwhile, and lets the running turn finish", async () => { + const working = await withSession(await seeded(), "running", TurnId.make("turn-1")); + const { model, refusal } = await apply(working, queue()); + expect(refusal).toBeNull(); + expect(threadOf(model).pendingMove).toMatchObject({ + requestId: REQUEST, + stage: "waiting", + mode: "when-done", + waitsFor: [{ code: "turn", words: "when this turn ends" }], + }); + + expect((await apply(model, turnStart())).refusal).toContain( + "waiting to move to Mac mini. Cancel the move to keep working here.", + ); + const sendWhenDone = { + type: "thread.follow-up.submit", + commandId: nextCommandId(), + threadId: THREAD, + turnId: TurnId.make("turn-1"), + message: { + messageId: MessageId.make("m-queued"), + role: "user", + text: "then this", + attachments: [], + }, + delivery: "queue", + createdAt: now, + } as OrchestrationCommand; + expect(threadMoveGuardRefusal(model, sendWhenDone)).toContain("waiting to move"); + // What the running turn needs still goes through. + for (const command of [ + { + type: "thread.turn.interrupt", + commandId: nextCommandId(), + threadId: THREAD, + createdAt: now, + }, + { + type: "thread.approval.respond", + commandId: nextCommandId(), + threadId: THREAD, + requestId: "approval-1", + decision: "accept", + createdAt: now, + }, + ] as OrchestrationCommand[]) { + expect(threadMoveGuardRefusal(model, command)).toBeNull(); + } + // A second move can't queue behind it, and a direct one can't jump it. + expect( + (await apply(model, queue("when-done", ThreadMoveRequestId.make("request-2")))).refusal, + ).toContain("already waiting"); + expect((await apply(model, startMove())).refusal).toContain("already waiting"); + }); + + it("refuses to queue what only the user can clear, even behind a running turn", async () => { + let model = await withSession(await seeded(), "running", TurnId.make("turn-1")); + ({ model } = await apply(model, { + type: "thread.pull-request-automation.set", + commandId: nextCommandId(), + threadId: THREAD, + pullRequestNumber: 294, + pullRequestUrl: "https://github.com/acme/widgets/pull/294", + autoFix: true, + })); + expect((await apply(model, queue())).refusal).toContain( + "Turn off pull request auto-fix and auto-merge", + ); + }); + + it("freezes only once its sessions are stopped, ending the queued move in the same decision", async () => { + let model = await withSession(await seeded(), "ready"); + ({ model } = await apply(model, queue())); + // Not quiesced yet: the fence isn't up. + expect((await apply(model, startQueued())).refusal).toContain("hasn't stopped yet"); + expect(await decided(model, quiesce())).toEqual([ + "thread.move-quiescing", + "thread.session-stop-requested", + ]); + ({ model } = await apply(model, quiesce())); + expect(threadOf(model).pendingMove?.stage).toBe("quiescing"); + // The session still shows as ready. + expect((await apply(model, startQueued())).refusal).toContain("hasn't stopped yet"); + // A start from an older queued move never goes ahead. + expect( + (await apply(model, startQueued(ThreadMoveRequestId.make("request-old")))).refusal, + ).toContain("cancelled or replaced"); + + model = await withSession(model, "stopped"); + expect(await decided(model, startQueued())).toEqual([ + "thread.move-queue-ended", + "thread.move-started", + ]); + ({ model } = await apply(model, startQueued())); + expect(threadOf(model).pendingMove ?? null).toBeNull(); + expect(threadOf(model).move).toMatchObject({ transferId: TRANSFER, queuedRequestId: REQUEST }); + }); + + it("stop and move now ends every runtime of the thread at once", async () => { + const running = await withSession(await seeded(), "running", TurnId.make("turn-1")); + const answering = { + ...running, + threads: running.threads.map((thread) => + thread.id === THREAD + ? ({ ...thread, sideTurn: { sideTurnId: SideTurnId.make("side-1") } } as typeof thread) + : thread, + ), + }; + expect(await decided(answering, queue("stop-now"))).toEqual([ + "thread.move-queued", + "thread.session-stop-requested", + "thread.side-turn-interrupt-requested", + ]); + const { model } = await apply(answering, queue("stop-now")); + expect(threadOf(model).pendingMove?.stage).toBe("quiescing"); + }); + + it("a cancel or an end naming an older request changes nothing", async () => { + let { model } = await apply(await seeded(), queue()); + const old = ThreadMoveRequestId.make("request-old"); + for (const command of [ + { + type: "thread.move.unqueue", + commandId: nextCommandId(), + threadId: THREAD, + requestId: old, + createdAt: now, + }, + { + type: "thread.move.queue-end", + commandId: nextCommandId(), + threadId: THREAD, + requestId: old, + reason: "blocked", + words: "This thread's agent set itself a time to come back. Cancel it first.", + createdAt: now, + }, + ] as OrchestrationCommand[]) { + expect(await decided(model, command)).toEqual([]); + } + expect(threadOf(model).pendingMove?.requestId).toBe(REQUEST); + + ({ model } = await apply(model, { + type: "thread.move.queue-end", + commandId: nextCommandId(), + threadId: THREAD, + requestId: REQUEST, + reason: "blocked", + words: "This thread's agent set itself a time to come back. Cancel it first.", + createdAt: now, + })); + expect(threadOf(model).pendingMove ?? null).toBeNull(); + expect(threadOf(model).activities.at(-1)).toMatchObject({ + kind: "thread.move-not-started", + summary: "Didn't move to Mac mini", + payload: { reason: "blocked", targetLabel: "Mac mini" }, + }); + // The thread works again. + expect((await apply(model, rename("Back to work"))).refusal).toBeNull(); + expect(threadMoveGuardRefusal(model, turnStart())).toBeNull(); + }); +}); diff --git a/apps/server/src/orchestration/decider.ts b/apps/server/src/orchestration/decider.ts index f72e8e43a..aaff1d995 100644 --- a/apps/server/src/orchestration/decider.ts +++ b/apps/server/src/orchestration/decider.ts @@ -112,7 +112,12 @@ import { decideMoveCancel, decideMoveComplete, decideMoveProgress, + decideMoveQueue, + decideMoveQueueEnd, + decideMoveQueueStatus, + decideMoveQuiesce, decideMoveStart, + decideMoveUnqueue, decideThreadImport, threadMoveGuardRefusal, } from "./threadMoveDecisions.ts"; @@ -598,6 +603,8 @@ function withArmedContinueAtReset( settingOn: boolean, ): OrchestrationSession { const resetsAt = session.lastErrorUsageLimit?.resetsAt ?? null; + // The user asked to move this thread: that wins over an automatic retry here. + if ((thread.pendingMove ?? null) !== null) return session; if ( resetsAt === null || session.continueAtReset != null || @@ -2301,6 +2308,49 @@ export const decideOrchestrationCommand = Effect.fn("decideOrchestrationCommand" }); } + // A move that waits for the thread to be free (docs/design/fleet.md, Part 3b). + case "thread.move.queue": { + const thread = yield* requireThread({ + readModel, + command, + threadId: command.threadId, + }); + const baseFor = childEventBase(command, command.createdAt); + const decided = decideMoveQueue(readModel, thread, command, baseFor); + // "Stop and move now" holds command output the way Stop does, so a + // command finishing later can't wake the agent the move stopped. + const held = + command.mode === "stop-now" && !("refusal" in decided) && decided.length > 0 + ? holdCommandOutputOnStop(thread, () => baseFor(thread.id), command.createdAt) + : []; + return yield* planChildThreadDecision( + command.type, + "refusal" in decided ? decided : [...decided, ...held], + ); + } + + case "thread.move.unqueue": + case "thread.move.quiesce": + case "thread.move.queue-status": + case "thread.move.queue-end": { + const thread = yield* requireThread({ + readModel, + command, + threadId: command.threadId, + }); + const baseFor = childEventBase(command, command.createdAt); + return yield* planChildThreadDecision( + command.type, + command.type === "thread.move.unqueue" + ? decideMoveUnqueue(thread, command, baseFor) + : command.type === "thread.move.quiesce" + ? decideMoveQuiesce(readModel, thread, command, baseFor) + : command.type === "thread.move.queue-status" + ? decideMoveQueueStatus(thread, command, baseFor) + : decideMoveQueueEnd(thread, command, baseFor), + ); + } + case "thread.move.start": case "thread.move.progress": case "thread.move.cancel": diff --git a/apps/server/src/orchestration/pendingMoveProjection.ts b/apps/server/src/orchestration/pendingMoveProjection.ts new file mode 100644 index 000000000..1e39ebbd8 --- /dev/null +++ b/apps/server/src/orchestration/pendingMoveProjection.ts @@ -0,0 +1,35 @@ +/** + * A thread's queued move (ThreadPendingMove, docs/design/fleet.md, Part 3b) + * after one of its events: one rule for the in-memory projector and the + * SQLite projection, so the two never disagree. Every event names the + * request it is about; one for another request leaves the pending move as it + * is. + */ +import type { OrchestrationEvent, ThreadPendingMove } from "@threadlines/contracts"; + +export type PendingMoveEvent = Extract< + OrchestrationEvent, + { + readonly type: + | "thread.move-queued" + | "thread.move-quiescing" + | "thread.move-queue-status-set" + | "thread.move-queue-ended"; + } +>; + +export function nextPendingMove( + current: ThreadPendingMove | null, + event: PendingMoveEvent, +): ThreadPendingMove | null { + if (event.type === "thread.move-queued") return event.payload.pendingMove; + if (current === null || current.requestId !== event.payload.requestId) return current; + switch (event.type) { + case "thread.move-quiescing": + return { ...current, stage: "quiescing" }; + case "thread.move-queue-status-set": + return { ...current, waitsFor: event.payload.waitsFor, hold: event.payload.hold }; + case "thread.move-queue-ended": + return null; + } +} diff --git a/apps/server/src/orchestration/projector.ts b/apps/server/src/orchestration/projector.ts index 581dae02b..5405d4fec 100644 --- a/apps/server/src/orchestration/projector.ts +++ b/apps/server/src/orchestration/projector.ts @@ -35,6 +35,7 @@ import { withUnqueuedAgentMessage, } from "@threadlines/shared/roomAgentRequests"; import { applyAgentPagePublication } from "@threadlines/shared/agentPages"; +import { nextPendingMove } from "./pendingMoveProjection.ts"; import { applyCommandRequestDismissed, applyCommandRequested, @@ -1038,11 +1039,29 @@ export function projectEvent( phase: "exporting", cancelRequested: false, startedAt: payload.startedAt, + ...(payload.queuedRequestId !== undefined + ? { queuedRequestId: payload.queuedRequestId } + : {}), }, }), })), ); + // A move waiting for the thread to be free (ThreadPendingMove). + case "thread.move-queued": + case "thread.move-quiescing": + case "thread.move-queue-status-set": + case "thread.move-queue-ended": { + const thread = nextBase.threads.find((entry) => entry.id === event.payload.threadId); + if (thread === undefined) return Effect.succeed(nextBase); + return Effect.succeed({ + ...nextBase, + threads: updateThread(nextBase.threads, thread.id, { + pendingMove: nextPendingMove(thread.pendingMove ?? null, event), + }), + }); + } + case "thread.move-progressed": return decodeForEvent(ThreadMoveProgressedPayload, event.payload, event.type, "payload").pipe( Effect.map((payload) => diff --git a/apps/server/src/orchestration/threadMoveDecisions.ts b/apps/server/src/orchestration/threadMoveDecisions.ts index 003819fbe..6cb4c8b2b 100644 --- a/apps/server/src/orchestration/threadMoveDecisions.ts +++ b/apps/server/src/orchestration/threadMoveDecisions.ts @@ -15,8 +15,17 @@ import { type OrchestrationReadModel, type OrchestrationThread, type ThreadId, + ThreadMoveNotStartedActivityKind, + type ThreadMoveNotStartedPayload, + type ThreadMoveWait, + type ThreadMoveWaitCode, } from "@threadlines/contracts"; import { threadPullRequestAutomationArmed } from "@threadlines/shared/linkedPullRequests"; +import { + APPROVAL_ACTIVITY_KINDS, + collectOpenPendingRequests, + countPendingUserInputs, +} from "@threadlines/shared/pendingRequests"; import { isRoomThread } from "@threadlines/shared/threadParticipants"; import { childHasLiveWork } from "./childThreadDecisions.ts"; @@ -37,32 +46,113 @@ export type ThreadMoveDecision = ReadonlyArray | { readonly refusa const refuse = (refusal: string) => ({ refusal }); +/** Something only the user can clear stops this thread from moving. */ +export type ThreadMoveBlockCode = + | "deleted" + | "archived" + | "moving" + | "moved" + | "general_chat" + | "room" + | "remote_origin" + | "child" + | "has_children" + | "queued_messages" + | "plan" + | "goal" + | "check_back" + | "command_request" + | "pull_request_automation"; + /** - * Why `thread` can't move right now, in words for the user, or null when it - * can. The same check runs in `moves.preview` (so the dialog explains before - * anyone clicks) and in the freeze decision. + * One reason a thread can't move right now. A `wait` goes away on its own + * (the turn ends, the user answers); a `blocked` needs the user. */ -export function threadMoveRefusal( +export type ThreadMoveRefusalEntry = + | { readonly kind: "wait"; readonly code: ThreadMoveWaitCode; readonly words: string } + | { readonly kind: "blocked"; readonly code: ThreadMoveBlockCode; readonly words: string }; + +/** The words a wait finishes "Moves to …" with. */ +const WAIT_WORDS: Record = { + turn: "when this turn ends", + approval: "once you answer", + question: "once you answer", + side_turn: "when its side answer finishes", + background: "when its background work finishes", + command_run: "when its command finishes running", + workflow_run: "when its workflow finishes", + voice: "when the voice chat ends", + shared_checkout: "when the other thread working in its folder finishes", +}; + +const BUSY_WORDS = "Wait for this thread to finish what it's doing, then move it."; +const SHARED_CHECKOUT_WORDS = + "Another thread is working in the same folder. Wait for it to finish, then move this one."; + +const wait = (code: ThreadMoveWaitCode): ThreadMoveRefusalEntry => ({ + kind: "wait", + code, + words: WAIT_WORDS[code], +}); +const blocked = (code: ThreadMoveBlockCode, words: string): ThreadMoveRefusalEntry => ({ + kind: "blocked", + code, + words, +}); + +/** A thread's checkout: its worktree, else its project's folder. */ +function checkoutOf(readModel: OrchestrationReadModel, thread: OrchestrationThread) { + return ( + thread.worktreePath ?? + readModel.projects.find((entry) => entry.id === thread.projectId)?.workspaceRoot ?? + null + ); +} + +/** + * Every reason `thread` can't move right now (docs/design/fleet.md, Parts 3 + * and 3b), each as a `wait` or a `blocked` with words for the user. Every + * check runs, so a busy thread doesn't hide a goal behind its running turn. + * The same check runs in `moves.preview` (so the dialog explains before + * anyone clicks), in `thread.move.queue`, and in the freeze decision. A + * pending move isn't one of them: each caller decides about that itself. + */ +export function threadMoveRefusals( readModel: OrchestrationReadModel, thread: OrchestrationThread, -): string | null { - if (thread.deletedAt !== null) return "This thread was deleted."; - if (thread.archivedAt !== null) return "Archived threads can't move. Unarchive it first."; - if ((thread.move ?? null) !== null) { - return thread.move?.movedAway !== undefined - ? `This thread already moved to ${thread.move.target.label}.` - : `This thread is already moving to ${thread.move?.target.label}.`; +): ReadonlyArray { + const found: ThreadMoveRefusalEntry[] = []; + if (thread.deletedAt !== null) found.push(blocked("deleted", "This thread was deleted.")); + if (thread.archivedAt !== null) { + found.push(blocked("archived", "Archived threads can't move. Unarchive it first.")); + } + const move = thread.move ?? null; + if (move !== null) { + found.push( + move.movedAway !== undefined + ? blocked("moved", `This thread already moved to ${move.target.label}.`) + : blocked("moving", `This thread is already moving to ${move.target.label}.`), + ); } const project = readModel.projects.find((candidate) => candidate.id === thread.projectId); - if (project?.kind === "general-chat") return "General chats can't move yet."; + if (project?.kind === "general-chat") { + found.push(blocked("general_chat", "General chats can't move yet.")); + } if (isRoomThread(thread) || thread.agentRequests.open.length > 0) { - return "Threads with more than one agent can't move yet."; + found.push(blocked("room", "Threads with more than one agent can't move yet.")); } if ((thread.remoteOrigin ?? null) !== null) { - return "This thread was started from another computer, so it stays here."; + found.push( + blocked("remote_origin", "This thread was started from another computer, so it stays here."), + ); } if (thread.parentThreadId !== null && thread.attachedToParent) { - return "This thread belongs to the thread that started it. Make it its own thread first."; + found.push( + blocked( + "child", + "This thread belongs to the thread that started it. Make it its own thread first.", + ), + ); } const hasLocalChildren = readModel.threads.some( (candidate) => @@ -70,31 +160,37 @@ export function threadMoveRefusal( candidate.attachedToParent && candidate.deletedAt === null, ); - if (hasLocalChildren || (thread.childRequests.remoteChildren ?? []).length > 0) { - return "Threads that started other threads can't move yet."; + if ( + hasLocalChildren || + (thread.childRequests.remoteChildren ?? []).length > 0 || + thread.childRequests.open.length > 0 + ) { + found.push(blocked("has_children", "Threads that started other threads can't move yet.")); } - const session = thread.session; + // Queued follow-ups would start the next turn the moment this one ends. + if ((thread.queuedFollowUps ?? []).length > 0) { + found.push(blocked("queued_messages", "Send or remove your queued messages first.")); + } + // Only the user's answer clears a plan, and implementing it starts a turn. + const latestTurnId = thread.latestTurn?.turnId ?? null; if ( - childHasLiveWork(thread) || - (session?.pendingBackgroundTaskCount ?? 0) > 0 || - (session?.awaitedBackgroundTaskCount ?? 0) > 0 + thread.proposedPlans.some( + (plan) => + latestTurnId !== null && + plan.turnId === latestTurnId && + plan.implementedAt === null && + plan.dismissedAt === null, + ) ) { - return "Wait for this thread to finish what it's doing, then move it."; + found.push(blocked("plan", "Implement or dismiss the plan first.")); } if (thread.goal !== null && thread.goal.status !== "complete") { - return "This thread has a goal set. Clear it first."; + found.push(blocked("goal", "This thread has a goal set. Clear it first.")); } if ((thread.checkBacks ?? []).length > 0) { - return "This thread's agent set itself a time to come back. Cancel it first."; - } - if ( - (thread.commandRuns ?? []).some((run) => - ["starting", "running", "waiting"].includes(run.status), - ) || - (thread.workflowRuns ?? []).some((run) => run.status === "running") || - thread.voiceActive === true - ) { - return "Wait for this thread to finish what it's doing, then move it."; + found.push( + blocked("check_back", "This thread's agent set itself a time to come back. Cancel it first."), + ); } const runRequestIds = new Set( (thread.commandRuns ?? []).flatMap((run) => @@ -106,27 +202,97 @@ export function threadMoveRefusal( (request) => request.dismissedAt === null && !runRequestIds.has(request.requestId), ) ) { - return "A command in this thread is waiting for you to run or dismiss it. Do that first."; + found.push( + blocked( + "command_request", + "A command in this thread is waiting for you to run or dismiss it. Do that first.", + ), + ); } if (threadPullRequestAutomationArmed(thread)) { - return "Turn off pull request auto-fix and auto-merge for this thread first."; + found.push( + blocked( + "pull_request_automation", + "Turn off pull request auto-fix and auto-merge for this thread first.", + ), + ); + } + + // What goes away on its own. + const session = thread.session; + if ( + thread.latestTurn?.state === "running" || + session?.status === "running" || + session?.status === "starting" + ) { + found.push(wait("turn")); + } + if (collectOpenPendingRequests(thread.activities, APPROVAL_ACTIVITY_KINDS).length > 0) { + found.push(wait("approval")); + } + if (countPendingUserInputs(thread.activities).pendingUserInputCount > 0) { + found.push(wait("question")); + } + if ((thread.sideTurn ?? null) !== null) found.push(wait("side_turn")); + if ( + (session?.pendingBackgroundTaskCount ?? 0) > 0 || + (session?.awaitedBackgroundTaskCount ?? 0) > 0 + ) { + found.push(wait("background")); + } + if ( + (thread.commandRuns ?? []).some((run) => + ["starting", "running", "waiting"].includes(run.status), + ) + ) { + found.push(wait("command_run")); } + if ((thread.workflowRuns ?? []).some((run) => run.status === "running")) { + found.push(wait("workflow_run")); + } + if (thread.voiceActive === true) found.push(wait("voice")); // Another thread at work in the same checkout could change the files while // they are being packed. - const checkout = thread.worktreePath ?? project?.workspaceRoot ?? null; + const checkout = checkoutOf(readModel, thread); const sharedCheckoutBusy = readModel.threads.some( (candidate) => candidate.id !== thread.id && candidate.deletedAt === null && - (candidate.worktreePath ?? - readModel.projects.find((entry) => entry.id === candidate.projectId)?.workspaceRoot ?? - null) === checkout && + checkoutOf(readModel, candidate) === checkout && childHasLiveWork(candidate), ); - if (sharedCheckoutBusy) { - return "Another thread is working in the same folder. Wait for it to finish, then move this one."; - } - return null; + if (sharedCheckoutBusy) found.push(wait("shared_checkout")); + return found; +} + +/** The waits among a thread's move refusals, as clients see them. */ +export function moveWaitsOf( + refusals: ReadonlyArray, +): ReadonlyArray { + return refusals.flatMap((entry) => + entry.kind === "wait" ? [{ code: entry.code, words: entry.words }] : [], + ); +} + +/** The first thing only the user can clear, or null. */ +export function firstMoveBlock(refusals: ReadonlyArray) { + return refusals.find((entry) => entry.kind === "blocked") ?? null; +} + +/** + * Why `thread` can't move straight away, in words for the user, or null when + * it can: a move that doesn't wait (`moves.start`) refuses waits too. + */ +export function threadMoveRefusal( + readModel: OrchestrationReadModel, + thread: OrchestrationThread, +): string | null { + const refusals = threadMoveRefusals(readModel, thread); + const block = firstMoveBlock(refusals); + if (block !== null) return block.words; + const first = refusals[0]; + if (first === undefined) return null; + return first.code === "shared_checkout" ? SHARED_CHECKOUT_WORDS : BUSY_WORDS; } /** @@ -170,47 +336,138 @@ function frozenWords(thread: OrchestrationThread): string { } /** - * Why `command` must not touch a moving or moved thread, or null. Covers the - * thread the command is for and a parent it would attach a thread to; a fork - * only reads its source, so it stays allowed (it is the honest way to keep - * working here). + * Commands that start work on a thread (docs/design/fleet.md, Part 3b, "The + * guard"). A thread with a pending move refuses them, so the wait stays + * short; what a running turn needs (approvals, answers, activity, session + * updates, interrupt) still goes through. Some only start work in one form: + * see `startsWork`. + */ +const WORK_START_COMMANDS: ReadonlySet = new Set([ + "thread.turn.start", + "thread.turn.retry", + "thread.turn.resend", + // Both steer and queue: "Send when done", outside assistants' queued + // messages and scheduled runs into an existing thread all come this way. + "thread.follow-up.submit", + "thread.side-turn.start", + "thread.realtime.start", + "thread.command-run.start", + "thread.goal.set", + "thread.participant.add", + "thread.context-compact.request", + // A first send on a thread made without one. + "thread.start", + // A continue at the usage limit's reset is a turn the user didn't just ask for. + "thread.continue-at-reset.fire", + "thread.pull-request-automation.set", + "thread.continue-at-reset.set", +]); + +/** Whether `command`, on its own thread, starts work there. */ +function startsWork(command: OrchestrationCommand): boolean { + if (!WORK_START_COMMANDS.has(command.type)) return false; + switch (command.type) { + // Turning a switch off is always allowed. + case "thread.pull-request-automation.set": + return ( + command.autoFix === true || (command.autoMerge !== undefined && command.autoMerge !== null) + ); + case "thread.continue-at-reset.set": + return command.enabled; + default: + return true; + } +} + +/** The words for a thread whose move waits. */ +function pendingWords(thread: OrchestrationThread): string { + const label = thread.pendingMove?.target.label ?? "another computer"; + return `This thread is waiting to move to ${label}. Cancel the move to keep working here.`; +} + +/** + * Why `command` must not touch a moving or moved thread, or start work on a + * thread whose move waits, or null. Covers the thread the command is for and + * a parent it would attach a thread to; a fork only reads its source, so it + * stays allowed (it is the honest way to keep working here). */ export function threadMoveGuardRefusal( readModel: OrchestrationReadModel, command: OrchestrationCommand, ): string | null { - const frozen = (threadId: ThreadId | null | undefined) => { - if (threadId === null || threadId === undefined) return null; - const thread = readModel.threads.find((candidate) => candidate.id === threadId); - return thread !== undefined && (thread.move ?? null) !== null ? thread : null; - }; + const find = (threadId: ThreadId | null | undefined) => + threadId === null || threadId === undefined + ? undefined + : readModel.threads.find((candidate) => candidate.id === threadId); + /** Why the thread can't take a new child or attach one, or null. */ + const parentRefusal = (thread: OrchestrationThread | undefined) => + thread === undefined + ? null + : (thread.move ?? null) !== null + ? frozenWords(thread) + : (thread.pendingMove ?? null) !== null + ? pendingWords(thread) + : null; if ("threadId" in command && command.type !== "thread.import") { - const thread = frozen(command.threadId); - if (thread !== null && !FROZEN_THREAD_COMMANDS.has(command.type)) { - return frozenWords(thread); + const thread = find(command.threadId); + if (thread !== undefined && (thread.move ?? null) !== null) { + if (!FROZEN_THREAD_COMMANDS.has(command.type)) return frozenWords(thread); + } else if ( + thread !== undefined && + (thread.pendingMove ?? null) !== null && + startsWork(command) + ) { + return pendingWords(thread); } } if (command.type === "thread.create") { - const parent = frozen(command.parentThreadId); - if (parent !== null) return frozenWords(parent); + const refusal = parentRefusal(find(command.parentThreadId)); + if (refusal !== null) return refusal; } if (command.type === "thread.parent-attachment.set" && command.attached) { - const child = readModel.threads.find((candidate) => candidate.id === command.threadId); - const parent = frozen(child?.parentThreadId); - if (parent !== null) return frozenWords(parent); + const refusal = parentRefusal(find(find(command.threadId)?.parentThreadId)); + if (refusal !== null) return refusal; } return null; } -/** The freeze: the thread now moves, and its session stops so nothing can start a turn. */ +/** Whether the thread's agent has no session left, as projected. */ +const sessionStopped = (thread: OrchestrationThread) => + thread.session === null || thread.session.status === "stopped"; + +/** + * The freeze: the thread now moves, and its session stops so nothing can + * start a turn. A move from the queue (`queuedRequestId`) must be the + * thread's current queued move, and starts only once the thread is free and + * its sessions are gone; its pending move ends in the same decision, so the + * thread is never between the two. + */ export function decideMoveStart( readModel: OrchestrationReadModel, thread: OrchestrationThread, command: CommandOf<"thread.move.start">, baseFor: EventBase, ): ThreadMoveDecision { - const refusal = threadMoveRefusal(readModel, thread); - if (refusal !== null) return refuse(refusal); + const pending = thread.pendingMove ?? null; + const queuedRequestId = command.queuedRequestId; + if (queuedRequestId === undefined) { + if (pending !== null) { + return refuse(`A move to ${pending.target.label} is already waiting. Cancel it first.`); + } + const refusal = threadMoveRefusal(readModel, thread); + if (refusal !== null) return refuse(refusal); + } else { + if (pending === null || pending.requestId !== queuedRequestId) { + return refuse("That queued move was cancelled or replaced."); + } + const refusals = threadMoveRefusals(readModel, thread); + const block = firstMoveBlock(refusals); + if (block !== null) return refuse(block.words); + if (refusals.length > 0) return refuse(BUSY_WORDS); + if (pending.stage !== "quiescing" || !sessionStopped(thread)) { + return refuse("This thread's agent hasn't stopped yet."); + } + } const started: PlannedEvent = { ...baseFor(thread.id), type: "thread.move-started", @@ -220,8 +477,24 @@ export function decideMoveStart( target: command.target, agent: command.agent, startedAt: command.createdAt, + ...(queuedRequestId !== undefined ? { queuedRequestId } : {}), }, }; + const queueEnded: PlannedEvent[] = + queuedRequestId === undefined + ? [] + : [ + { + ...baseFor(thread.id), + type: "thread.move-queue-ended", + payload: { + threadId: thread.id, + requestId: queuedRequestId, + reason: "started", + endedAt: command.createdAt, + }, + }, + ]; // Settings cards waiting for an answer don't move: they end here, with why. const cancelled = cancelWaitingSettingsChanges( thread, @@ -229,8 +502,8 @@ export function decideMoveStart( command.createdAt, baseFor, ); - if (thread.session === null || thread.session.status === "stopped") { - return [...cancelled, started]; + if (sessionStopped(thread)) { + return [...cancelled, ...queueEnded, started]; } return [ ...cancelled, @@ -243,6 +516,221 @@ export function decideMoveStart( ]; } +/** The thread's pending move, when it is the one `requestId` names. */ +function currentPendingMove(thread: OrchestrationThread, requestId: string) { + const pending = thread.pendingMove ?? null; + return pending !== null && pending.requestId === requestId ? pending : null; +} + +/** Ending every runtime of the thread: "Stop and move now", and the barrier before a freeze. */ +function sessionStopEvents( + thread: OrchestrationThread, + createdAt: string, + baseFor: EventBase, +): PlannedEvent[] { + const sideTurn = thread.sideTurn ?? null; + return [ + { + ...baseFor(thread.id), + type: "thread.session-stop-requested", + payload: { threadId: thread.id, createdAt }, + }, + // A side answer runs in a runtime of its own. + ...(sideTurn !== null + ? [ + { + ...baseFor(thread.id), + type: "thread.side-turn-interrupt-requested" as const, + payload: { threadId: thread.id, sideTurnId: sideTurn.sideTurnId, createdAt }, + }, + ] + : []), + ]; +} + +/** + * Queues a move that starts once the thread is free (docs/design/fleet.md, + * Part 3b). Refused when something only the user can clear stands in the + * way, or another move waits or runs; allowed with any number of waits, and + * with none (the queue then starts it at once). "Stop and move now" ends the + * thread's sessions in the same decision; the caller adds what Stop holds + * (command output that would wake the agent). + */ +export function decideMoveQueue( + readModel: OrchestrationReadModel, + thread: OrchestrationThread, + command: CommandOf<"thread.move.queue">, + baseFor: EventBase, +): ThreadMoveDecision { + const pending = thread.pendingMove ?? null; + if (pending !== null) { + // The same request again (a retried call) is no news. + return pending.requestId === command.requestId + ? [] + : refuse(`A move to ${pending.target.label} is already waiting. Cancel it first.`); + } + const refusals = threadMoveRefusals(readModel, thread); + const block = firstMoveBlock(refusals); + if (block !== null) return refuse(block.words); + const queued: PlannedEvent = { + ...baseFor(thread.id), + type: "thread.move-queued", + payload: { + threadId: thread.id, + pendingMove: { + requestId: command.requestId, + target: command.target, + agent: command.agent, + mode: command.mode, + stage: command.mode === "stop-now" ? "quiescing" : "waiting", + requestedAt: command.createdAt, + requestedBy: command.requestedBy, + waitsFor: moveWaitsOf(refusals), + hold: null, + }, + }, + }; + return command.mode === "stop-now" + ? [queued, ...sessionStopEvents(thread, command.createdAt, baseFor)] + : [queued]; +} + +/** Cancel: the queued move `requestId` ends; a cancel for an older one changes nothing. */ +export function decideMoveUnqueue( + thread: OrchestrationThread, + command: CommandOf<"thread.move.unqueue">, + baseFor: EventBase, +): ThreadMoveDecision { + if (currentPendingMove(thread, command.requestId) === null) return []; + return [ + { + ...baseFor(thread.id), + type: "thread.move-queue-ended", + payload: { + threadId: thread.id, + requestId: command.requestId, + reason: "cancelled", + endedAt: command.createdAt, + }, + }, + ]; +} + +/** + * The barrier's first step: the thread is free, so its sessions end, and the + * session fence holds from here (`stage: "quiescing"`). Asked again while + * quiescing, it only stops a session the projection still shows. + */ +export function decideMoveQuiesce( + readModel: OrchestrationReadModel, + thread: OrchestrationThread, + command: CommandOf<"thread.move.quiesce">, + baseFor: EventBase, +): ThreadMoveDecision { + const pending = currentPendingMove(thread, command.requestId); + if (pending === null) return []; + const refusals = threadMoveRefusals(readModel, thread); + const block = firstMoveBlock(refusals); + if (block !== null) return refuse(block.words); + if (refusals.length > 0) return refuse(BUSY_WORDS); + if (pending.stage === "quiescing") { + return sessionStopped(thread) ? [] : sessionStopEvents(thread, command.createdAt, baseFor); + } + return [ + { + ...baseFor(thread.id), + type: "thread.move-quiescing", + payload: { threadId: thread.id, requestId: command.requestId, updatedAt: command.createdAt }, + }, + ...sessionStopEvents(thread, command.createdAt, baseFor), + ]; +} + +const sameWaits = (left: ReadonlyArray, right: ReadonlyArray) => + left.length === right.length && + left.every( + (entry, index) => entry.code === right[index]?.code && entry.words === right[index].words, + ); + +/** What the queued move waits for now, and what holds it; unchanged is no news. */ +export function decideMoveQueueStatus( + thread: OrchestrationThread, + command: CommandOf<"thread.move.queue-status">, + baseFor: EventBase, +): ThreadMoveDecision { + const pending = currentPendingMove(thread, command.requestId); + if (pending === null) return []; + const hold = pending.hold ?? null; + if ( + sameWaits(pending.waitsFor ?? [], command.waitsFor) && + hold?.kind === command.hold?.kind && + hold?.words === command.hold?.words + ) { + return []; + } + return [ + { + ...baseFor(thread.id), + type: "thread.move-queue-status-set", + payload: { + threadId: thread.id, + requestId: command.requestId, + waitsFor: command.waitsFor, + hold: command.hold, + updatedAt: command.createdAt, + }, + }, + ]; +} + +/** + * The queued move can't go (or its thread is gone): it ends, and the thread + * says why ("Didn't move: …") until the user dismisses it. + */ +export function decideMoveQueueEnd( + thread: OrchestrationThread, + command: CommandOf<"thread.move.queue-end">, + baseFor: EventBase, +): ThreadMoveDecision { + const pending = currentPendingMove(thread, command.requestId); + if (pending === null) return []; + const payload: ThreadMoveNotStartedPayload = { + requestId: command.requestId, + targetLabel: pending.target.label, + reason: command.reason, + words: command.words, + }; + return [ + { + ...baseFor(thread.id), + type: "thread.move-queue-ended", + payload: { + threadId: thread.id, + requestId: command.requestId, + reason: command.reason, + words: command.words, + endedAt: command.createdAt, + }, + }, + { + ...baseFor(thread.id), + type: "thread.activity-appended", + payload: { + threadId: thread.id, + activity: { + id: EventId.make(`move-not-started:${command.requestId}`), + tone: "error", + kind: ThreadMoveNotStartedActivityKind, + summary: `Didn't move to ${pending.target.label}`, + payload, + turnId: null, + createdAt: command.createdAt, + }, + }, + }, + ]; +} + /** The move named by `transferId`, when it is the thread's current one and still under way. */ function currentMove(thread: OrchestrationThread, transferId: string) { const move = thread.move ?? null; diff --git a/apps/server/src/persistence/Layers/ProjectionThreads.ts b/apps/server/src/persistence/Layers/ProjectionThreads.ts index b41fecdd2..1edbf0454 100644 --- a/apps/server/src/persistence/Layers/ProjectionThreads.ts +++ b/apps/server/src/persistence/Layers/ProjectionThreads.ts @@ -29,6 +29,7 @@ import { OrchestrationSettingsChangeRequest, ThreadId, OrchestrationThreadMove, + ThreadPendingMove, ThreadMovedFrom, ThreadRemoteOrigin, OrchestrationOutsideWork, @@ -63,6 +64,7 @@ const ProjectionThreadDbRow = ProjectionThread.mapFields( outsideOrigin: Schema.NullOr(Schema.fromJsonString(OutsideAgentRef)), outsideWork: Schema.NullOr(Schema.fromJsonString(Schema.Array(OrchestrationOutsideWork))), move: Schema.NullOr(Schema.fromJsonString(OrchestrationThreadMove)), + pendingMove: Schema.NullOr(Schema.fromJsonString(ThreadPendingMove)), movedFrom: Schema.NullOr(Schema.fromJsonString(ThreadMovedFrom)), roomContext: Schema.fromJsonString( Schema.Record(Schema.String, OrchestrationRoomContextCursor), @@ -125,6 +127,7 @@ const makeProjectionThreadRepository = Effect.gen(function* () { outside_origin_json, outside_work_json, move_json, + pending_move_json, moved_from_json, history_seed_owed, history_seed_message_id, @@ -192,6 +195,7 @@ const makeProjectionThreadRepository = Effect.gen(function* () { ${row.outsideOrigin ? JSON.stringify(row.outsideOrigin) : null}, ${row.outsideWork && row.outsideWork.length > 0 ? JSON.stringify(row.outsideWork) : null}, ${row.move ? JSON.stringify(row.move) : null}, + ${row.pendingMove ? JSON.stringify(row.pendingMove) : null}, ${row.movedFrom ? JSON.stringify(row.movedFrom) : null}, ${row.historySeedOwed ?? 0}, ${row.historySeedMessageId ?? null}, @@ -251,6 +255,7 @@ const makeProjectionThreadRepository = Effect.gen(function* () { outside_origin_json = excluded.outside_origin_json, outside_work_json = excluded.outside_work_json, move_json = excluded.move_json, + pending_move_json = excluded.pending_move_json, moved_from_json = excluded.moved_from_json, history_seed_owed = excluded.history_seed_owed, history_seed_message_id = excluded.history_seed_message_id, @@ -317,6 +322,7 @@ const makeProjectionThreadRepository = Effect.gen(function* () { outside_origin_json AS "outsideOrigin", outside_work_json AS "outsideWork", move_json AS "move", + pending_move_json AS "pendingMove", moved_from_json AS "movedFrom", history_seed_owed AS "historySeedOwed", history_seed_message_id AS "historySeedMessageId", @@ -385,6 +391,7 @@ const makeProjectionThreadRepository = Effect.gen(function* () { outside_origin_json AS "outsideOrigin", outside_work_json AS "outsideWork", move_json AS "move", + pending_move_json AS "pendingMove", moved_from_json AS "movedFrom", history_seed_owed AS "historySeedOwed", history_seed_message_id AS "historySeedMessageId", diff --git a/apps/server/src/persistence/Migrations.ts b/apps/server/src/persistence/Migrations.ts index 87c1181cd..cdeb5f818 100644 --- a/apps/server/src/persistence/Migrations.ts +++ b/apps/server/src/persistence/Migrations.ts @@ -130,6 +130,7 @@ import Migration0107 from "./Migrations/107_ThemeEntryOffer.ts"; import Migration0108 from "./Migrations/108_ThemePick.ts"; import Migration0109 from "./Migrations/109_ThemeShowcaseTakenBack.ts"; import Migration0110 from "./Migrations/110_ThreadWorktreeOrigin.ts"; +import Migration0111 from "./Migrations/111_ThreadPendingMove.ts"; /** * Migration loader with all migrations defined inline. @@ -252,6 +253,7 @@ export const migrationEntries = [ [108, "ThemePick", Migration0108], [109, "ThemeShowcaseTakenBack", Migration0109], [110, "ThreadWorktreeOrigin", Migration0110], + [111, "ThreadPendingMove", Migration0111], ] as const; /** Highest id in the full registry, so the "n of total" count is the real total. */ diff --git a/apps/server/src/persistence/Migrations/111_ThreadPendingMove.ts b/apps/server/src/persistence/Migrations/111_ThreadPendingMove.ts new file mode 100644 index 000000000..390ef2a5b --- /dev/null +++ b/apps/server/src/persistence/Migrations/111_ThreadPendingMove.ts @@ -0,0 +1,14 @@ +import * as Effect from "effect/Effect"; +import * as SqlClient from "effect/sql/SqlClient"; + +/** + * A move waiting for its thread to be free (ThreadPendingMove, + * docs/design/fleet.md, Part 3b): `pending_move_json` on + * `projection_threads`, null for every thread without one. + * + * A new column only; no row is rewritten. + */ +export default Effect.gen(function* () { + const sql = yield* SqlClient.SqlClient; + yield* sql`ALTER TABLE projection_threads ADD COLUMN pending_move_json TEXT`; +}); diff --git a/apps/server/src/persistence/Services/ProjectionThreads.ts b/apps/server/src/persistence/Services/ProjectionThreads.ts index 595b29db6..b0d04990d 100644 --- a/apps/server/src/persistence/Services/ProjectionThreads.ts +++ b/apps/server/src/persistence/Services/ProjectionThreads.ts @@ -32,6 +32,7 @@ import { OrchestrationChildRequestState, OrchestrationSettingsChangeRequest, OrchestrationThreadMove, + ThreadPendingMove, ThreadMovedFrom, ThreadRemoteOrigin, OrchestrationOutsideWork, @@ -167,6 +168,8 @@ export const ProjectionThread = Schema.Struct({ outsideWork: Schema.optional(Schema.NullOr(Schema.Array(OrchestrationOutsideWork))), /** See `OrchestrationThread.move`. Absent or null reads as none. */ move: Schema.optional(Schema.NullOr(OrchestrationThreadMove)), + /** See `OrchestrationThread.pendingMove`. Absent or null reads as none. */ + pendingMove: Schema.optional(Schema.NullOr(ThreadPendingMove)), /** See `OrchestrationThread.movedFrom`. Absent or null reads as none. */ movedFrom: Schema.optional(Schema.NullOr(ThreadMovedFrom)), /** See `OrchestrationThread.historySeedOwed`, as 0 or 1. Absent or null reads as 0. */ From 10b2f4bb2c1c24d12d6b299dfbb8d7b045e4a3f3 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:53:04 -0500 Subject: [PATCH 04/48] feat(server): no session of a moving thread starts, and a stop can be proven ThreadSessionAdmission is asked before ProviderService starts or recovers any session of a thread, whatever its key; the real layer reads the projection and refuses while a queued move is quiescing or once the thread moves. ProviderService.quiesceThread stops every runtime of a thread and asks each driver to confirm its processes are gone (confirmThreadStopped), with an agent-process ledger drivers share: it notes what an agent started when its stop begins, waits for the agent to exit, and ends what outlived it. --- .../OrchestrationEngineHarness.integration.ts | 3 + .../providerService.integration.test.ts | 6 +- .../provider/Layers/ProviderService.test.ts | 17 ++ .../src/provider/Layers/ProviderService.ts | 76 +++++- .../Layers/ProviderSessionReaper.test.ts | 1 + .../provider/Layers/ThreadSessionAdmission.ts | 39 +++ .../src/provider/Services/ProviderAdapter.ts | 18 ++ .../src/provider/Services/ProviderService.ts | 17 +- .../Services/ThreadSessionAdmission.ts | 33 +++ .../server/src/provider/agentProcessLedger.ts | 241 ++++++++++++++++++ apps/server/src/server.ts | 3 + 11 files changed, 450 insertions(+), 4 deletions(-) create mode 100644 apps/server/src/provider/Layers/ThreadSessionAdmission.ts create mode 100644 apps/server/src/provider/Services/ThreadSessionAdmission.ts create mode 100644 apps/server/src/provider/agentProcessLedger.ts diff --git a/apps/server/integration/OrchestrationEngineHarness.integration.ts b/apps/server/integration/OrchestrationEngineHarness.integration.ts index 2177ea7e0..4f1da695d 100644 --- a/apps/server/integration/OrchestrationEngineHarness.integration.ts +++ b/apps/server/integration/OrchestrationEngineHarness.integration.ts @@ -83,6 +83,7 @@ import * as VcsDriverRegistry from "../src/vcs/VcsDriverRegistry.ts"; import { VcsStatusBroadcaster } from "../src/vcs/VcsStatusBroadcaster.ts"; import { GitWorkflowService } from "../src/git/GitWorkflowService.ts"; import * as VcsProcess from "../src/vcs/VcsProcess.ts"; +import { ThreadSessionAdmissionPermissive } from "../src/provider/Services/ThreadSessionAdmission.ts"; const decodeCodexSettings = Schema.decodeEffect(CodexSettings); @@ -284,12 +285,14 @@ export const makeOrchestrationIntegrationHarness = ( const providerEventLoggersLayer = Layer.succeed(ProviderEventLoggers, NoOpProviderEventLoggers); const providerLayer = useRealCodex ? makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(providerSessionDirectoryLayer), Layer.provide(realCodexRegistry), Layer.provide(AnalyticsService.layerTest), Layer.provide(providerEventLoggersLayer), ) : makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(providerSessionDirectoryLayer), Layer.provide(fakeRegistry!), Layer.provide(AnalyticsService.layerTest), diff --git a/apps/server/integration/providerService.integration.test.ts b/apps/server/integration/providerService.integration.test.ts index 847f2dd3d..a5072f6c5 100644 --- a/apps/server/integration/providerService.integration.test.ts +++ b/apps/server/integration/providerService.integration.test.ts @@ -37,6 +37,7 @@ import { codexTurnToolFixture, codexTurnTextFixture, } from "./fixtures/providerRuntime.ts"; +import { ThreadSessionAdmissionPermissive } from "../src/provider/Services/ThreadSessionAdmission.ts"; const codexInstanceId = ProviderInstanceId.make("codex"); @@ -74,7 +75,10 @@ const makeIntegrationFixture = Effect.gen(function* () { Layer.succeed(ProviderEventLoggers, NoOpProviderEventLoggers), ).pipe(Layer.provide(SqlitePersistenceMemory)); - const layer = makeProviderServiceLive().pipe(Layer.provide(shared)); + const layer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), + Layer.provide(shared), + ); return { cwd, diff --git a/apps/server/src/provider/Layers/ProviderService.test.ts b/apps/server/src/provider/Layers/ProviderService.test.ts index 96eb9521a..ff62b72ab 100644 --- a/apps/server/src/provider/Layers/ProviderService.test.ts +++ b/apps/server/src/provider/Layers/ProviderService.test.ts @@ -74,6 +74,7 @@ import { import { ServerSettingsService } from "../../serverSettings.ts"; import { AnalyticsService } from "../../telemetry/Services/AnalyticsService.ts"; import { makeAdapterRegistryMock } from "../testUtils/providerAdapterRegistryMock.ts"; +import { ThreadSessionAdmissionPermissive } from "../Services/ThreadSessionAdmission.ts"; const defaultServerSettingsLayer = ServerSettingsService.layerTest(); @@ -437,6 +438,7 @@ function makeProviderServiceLayer() { const layer = it.layer( Layer.mergeAll( makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(providerAdapterLayer), Layer.provide(directoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -480,6 +482,7 @@ it.effect("ProviderServiceLive catches stopAll failures during shutdown", () => const directoryLayer = ProviderSessionDirectoryLive.pipe(Layer.provide(runtimeRepositoryLayer)); const providerLayer = Layer.mergeAll( makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(providerAdapterLayer), Layer.provide(directoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -554,6 +557,7 @@ it.effect( Layer.provide(runtimeRepositoryLayer), ); const providerLayer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(providerAdapterLayer), Layer.provide(directoryLayer), Layer.provide(serverSettingsLayer), @@ -616,6 +620,7 @@ it.effect("ProviderServiceLive rejects new sessions for disabled custom instance ); const directoryLayer = ProviderSessionDirectoryLive.pipe(Layer.provide(runtimeRepositoryLayer)); const providerLayer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(providerAdapterLayer), Layer.provide(directoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -666,6 +671,7 @@ it.effect("ProviderServiceLive writes canonical events to the emitting thread se close: () => Effect.void, }, }).pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, registry)), Layer.provide(directoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -714,6 +720,7 @@ it.effect( Layer.provide(runtimeRepositoryLayer), ); const providerLayer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, registry)), Layer.provide(directoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -813,6 +820,7 @@ it.effect("ProviderServiceLive reports a failure under the category the agent na ), ); const providerLayer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, registry)), Layer.provide(directoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -886,6 +894,7 @@ const makeEarlyStartHarness = () => { ); const directoryLayer = ProviderSessionDirectoryLive.pipe(Layer.provide(runtimeRepositoryLayer)); const layer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, registry)), Layer.provide(directoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -1274,6 +1283,7 @@ it.effect("ProviderServiceLive keeps persisted resumable sessions on startup", ( }).pipe(Effect.provide(directoryLayer)); const providerLayer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, registry)), Layer.provide(directoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -1333,6 +1343,7 @@ it.effect( Layer.provide(runtimeRepositoryLayer), ); const firstProviderLayer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, firstRegistry)), Layer.provide(firstDirectoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -1385,6 +1396,7 @@ it.effect( Layer.provide(runtimeRepositoryLayer), ); const secondProviderLayer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, secondRegistry)), Layer.provide(secondDirectoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -2384,6 +2396,7 @@ routing.layer("ProviderServiceLive routing", (it) => { Layer.provide(runtimeRepositoryLayer), ); const firstProviderLayer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, firstRegistry)), Layer.provide(firstDirectoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -2415,6 +2428,7 @@ routing.layer("ProviderServiceLive routing", (it) => { Layer.provide(runtimeRepositoryLayer), ); const secondProviderLayer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, secondRegistry)), Layer.provide(secondDirectoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -2498,6 +2512,7 @@ routing.layer("ProviderServiceLive routing", (it) => { }; const providerLayerFor = (adapter: ReturnType["adapter"]) => makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, accountsRegistry(adapter))), Layer.provide(ProviderSessionDirectoryLive.pipe(Layer.provide(runtimeRepositoryLayer))), Layer.provide(defaultServerSettingsLayer), @@ -2560,6 +2575,7 @@ routing.layer("ProviderServiceLive routing", (it) => { Layer.provide(runtimeRepositoryLayer), ); const firstProviderLayer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, firstRegistry)), Layer.provide(firstDirectoryLayer), Layer.provide(defaultServerSettingsLayer), @@ -2586,6 +2602,7 @@ routing.layer("ProviderServiceLive routing", (it) => { Layer.provide(runtimeRepositoryLayer), ); const secondProviderLayer = makeProviderServiceLive().pipe( + Layer.provide(ThreadSessionAdmissionPermissive), Layer.provide(Layer.succeed(ProviderAdapterRegistry, secondRegistry)), Layer.provide(secondDirectoryLayer), Layer.provide(defaultServerSettingsLayer), diff --git a/apps/server/src/provider/Layers/ProviderService.ts b/apps/server/src/provider/Layers/ProviderService.ts index 84668c410..8d80b7662 100644 --- a/apps/server/src/provider/Layers/ProviderService.ts +++ b/apps/server/src/provider/Layers/ProviderService.ts @@ -67,6 +67,7 @@ import { } from "../../observability/Metrics.ts"; import { type ProviderAdapterError, ProviderValidationError } from "../Errors.ts"; import type { ProviderAdapterShape } from "../Services/ProviderAdapter.ts"; +import { ThreadSessionAdmission } from "../Services/ThreadSessionAdmission.ts"; import { ProviderAdapterRegistry } from "../Services/ProviderAdapterRegistry.ts"; import { ProviderService, type ProviderServiceShape } from "../Services/ProviderService.ts"; import { @@ -381,6 +382,7 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( const registry = yield* ProviderAdapterRegistry; const directory = yield* ProviderSessionDirectory; + const admission = yield* ThreadSessionAdmission; const runtimeEventPubSub = yield* PubSub.unbounded(); const nowIso = Effect.map(DateTime.now, DateTime.formatIso); const turnLatency = makeTurnLatencyTracker(); @@ -728,10 +730,22 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( () => reconcileInstanceSubscriptions, ).pipe(Effect.forkScoped); + /** + * The session fence (ThreadSessionAdmission): no session of a thread that + * is moving, moved, or having its sessions ended for a move starts or + * recovers, whatever its key and whoever asks. + */ + const requireAdmission = (operation: string, sessionKey: ThreadId) => + Effect.gen(function* () { + const refusal = yield* admission.refusal(parseSessionKey(sessionKey).threadId); + if (refusal !== null) return yield* toValidationError(operation, refusal); + }); + const recoverSessionForThread = Effect.fn("recoverSessionForThread")(function* (input: { readonly binding: ProviderRuntimeBinding; readonly operation: string; }) { + yield* requireAdmission(input.operation, input.binding.threadId); const bindingInstanceId = yield* requireBindingInstanceId(input.operation, input.binding); yield* Effect.annotateCurrentSpan({ "provider.operation": "recover-session", @@ -950,6 +964,7 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( schema: ProviderSessionStartInput, payload: rawInput, }); + yield* requireAdmission("ProviderService.startSession", threadId); const resolvedInstanceId = yield* requireBindingInstanceId( "ProviderService.startSession", @@ -2945,10 +2960,14 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( const input = { threadId }; let metricProvider = "unknown"; return yield* Effect.gen(function* () { + // A fenced thread's runtime isn't brought back just to delete its + // transcript: only its binding goes (and a live runtime, if any). + const admitted = + (yield* admission.refusal(parseSessionKey(input.threadId).threadId)) === null; const routed = yield* resolveRoutableSession({ threadId: input.threadId, operation: "ProviderService.deleteThread", - allowRecovery: true, + allowRecovery: admitted, }); turnLatency.sessionStopped(input.threadId); earlySessions.forget(input.threadId); @@ -2959,7 +2978,7 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( "provider.thread_id": input.threadId, "provider.native_delete_supported": routed.adapter.deleteThread !== undefined, }); - if (routed.adapter.deleteThread) { + if (routed.adapter.deleteThread && (admitted || routed.isActive)) { yield* routed.adapter.deleteThread(routed.threadId); } else if (routed.isActive) { yield* routed.adapter.stopSession(routed.threadId); @@ -2980,6 +2999,58 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( ); }); + const quiesceThread: ProviderServiceShape["quiesceThread"] = Effect.fn("quiesceThread")( + function* (threadId) { + const ofThread = (key: string) => parseSessionKey(ThreadId.make(key)).threadId === threadId; + const liveOfThread = Effect.forEach(yield* getAdapterEntries, ([instanceId, adapter]) => + adapter + .listSessions() + .pipe( + Effect.map((sessions) => + sessions + .filter((session) => session.status !== "closed" && ofThread(session.threadId)) + .map((session) => ({ instanceId, adapter, key: session.threadId })), + ), + ), + ).pipe(Effect.map((perAdapter) => perAdapter.flat())); + const live = yield* liveOfThread; + const ran = new Set(live.map((entry) => entry.instanceId)); + // Early starts still starting have no session to list yet. + const keys = new Set(live.map((entry) => entry.key)); + for (const key of earlyStartRuns.keys()) { + if (ofThread(key)) keys.add(ThreadId.make(key)); + } + const problems: string[] = []; + for (const key of keys) { + yield* stopSession({ threadId: key }).pipe( + Effect.catchCause((cause) => + Effect.sync(() => { + const error = Cause.squash(cause); + problems.push( + `stopping it failed (${error instanceof Error ? error.message : String(error)})`, + ); + }), + ), + ); + } + for (const entry of yield* liveOfThread) { + problems.push(`${entry.adapter.provider} still lists its session`); + } + // Each driver proves its processes, and what they started, are gone. + for (const [instanceId, adapter] of yield* getAdapterEntries) { + if (adapter.confirmThreadStopped === undefined) { + if (ran.has(instanceId)) problems.push(`${adapter.provider} can't confirm it stopped`); + continue; + } + const confirmation = yield* adapter.confirmThreadStopped(threadId); + if (!confirmation.confirmed) problems.push(confirmation.reason); + } + return problems.length === 0 + ? ({ confirmed: true } as const) + : ({ confirmed: false, reason: problems.join("; ") } as const); + }, + ); + const runStopAll = Effect.fn("runStopAll")(function* () { const threadIds = yield* directory.listThreadIds(); const currentAdapters = yield* getAdapterEntries; @@ -3078,6 +3149,7 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( respondToUserInput, stopSession, listSessions, + quiesceThread, getCapabilities, getInstanceInfo, rollbackConversation, diff --git a/apps/server/src/provider/Layers/ProviderSessionReaper.test.ts b/apps/server/src/provider/Layers/ProviderSessionReaper.test.ts index 9748a2e17..6f8e401ef 100644 --- a/apps/server/src/provider/Layers/ProviderSessionReaper.test.ts +++ b/apps/server/src/provider/Layers/ProviderSessionReaper.test.ts @@ -246,6 +246,7 @@ describe("ProviderSessionReaper", () => { respondToUserInput: () => unsupported(), stopSession, listSessions: () => Effect.succeed(input.activeProviderSessions ?? []), + quiesceThread: () => Effect.succeed({ confirmed: true } as const), getCapabilities: () => Effect.succeed({ sessionModelSwitch: "in-session" }), getInstanceInfo: (instanceId) => { const driverKind = ProviderDriverKind.make(String(instanceId)); diff --git a/apps/server/src/provider/Layers/ThreadSessionAdmission.ts b/apps/server/src/provider/Layers/ThreadSessionAdmission.ts new file mode 100644 index 000000000..f5bf4867b --- /dev/null +++ b/apps/server/src/provider/Layers/ThreadSessionAdmission.ts @@ -0,0 +1,39 @@ +/** + * The session fence as the projection has it (ThreadSessionAdmission): read + * from the thread's row at every start, so it holds across restarts and + * needs no cache to keep in step. + */ +import * as Effect from "effect/Effect"; +import * as Layer from "effect/Layer"; +import * as Option from "effect/Option"; + +import { ProjectionThreadRepository } from "../../persistence/Services/ProjectionThreads.ts"; +import { + ThreadSessionAdmission, + type ThreadSessionAdmissionShape, +} from "../Services/ThreadSessionAdmission.ts"; + +const make = Effect.gen(function* () { + const threads = yield* ProjectionThreadRepository; + const refusal: ThreadSessionAdmissionShape["refusal"] = (threadId) => + threads.getById({ threadId }).pipe( + Effect.map((row) => { + if (Option.isNone(row)) return null; + const move = row.value.move ?? null; + if (move !== null) { + return move.movedAway !== undefined + ? `This thread moved to ${move.target.label}, so its agent can't start here.` + : `This thread is moving to ${move.target.label}, so its agent can't start here.`; + } + const pending = row.value.pendingMove ?? null; + return pending !== null && pending.stage === "quiescing" + ? `This thread's agent is being stopped to move it to ${pending.target.label}.` + : null; + }), + // A thread that can't be read is never started on a guess. + Effect.orElseSucceed(() => "This thread couldn't be read, so its agent wasn't started."), + ); + return ThreadSessionAdmission.of({ refusal }); +}); + +export const ThreadSessionAdmissionLive = Layer.effect(ThreadSessionAdmission, make); diff --git a/apps/server/src/provider/Services/ProviderAdapter.ts b/apps/server/src/provider/Services/ProviderAdapter.ts index adf2bb7e4..36895d0d5 100644 --- a/apps/server/src/provider/Services/ProviderAdapter.ts +++ b/apps/server/src/provider/Services/ProviderAdapter.ts @@ -456,8 +456,26 @@ export interface ProviderAdapterShape { */ readonly stopAll: () => Effect.Effect; + /** + * Proves that nothing this adapter ran for a thread is still running + * (docs/design/fleet.md, Part 3b, "Confirmed teardown"): every runtime of + * every session key whose thread is `threadId` (its own agent, room + * agents, side answers, early starts), and what those runtimes started. + * Called after their sessions were stopped; never starts anything. Waits + * (bounded) for the agent's processes to exit and ends what outlived the + * stop. `unconfirmed` when it can't prove it, with why; never a guess. + * Absent: the driver can't confirm, which counts as unconfirmed for a + * thread it ran. + */ + readonly confirmThreadStopped?: (threadId: ThreadId) => Effect.Effect; + /** * Canonical runtime event stream emitted by this adapter. */ readonly streamEvents: Stream.Stream; } + +/** Whether a thread's agents are proven stopped (ProviderAdapterShape.confirmThreadStopped). */ +export type ProviderStopConfirmation = + | { readonly confirmed: true } + | { readonly confirmed: false; readonly reason: string }; diff --git a/apps/server/src/provider/Services/ProviderService.ts b/apps/server/src/provider/Services/ProviderService.ts index b15be5b42..20e48e502 100644 --- a/apps/server/src/provider/Services/ProviderService.ts +++ b/apps/server/src/provider/Services/ProviderService.ts @@ -56,7 +56,11 @@ import type * as Effect from "effect/Effect"; import type * as Stream from "effect/Stream"; import type { ProviderAdapterError, ProviderServiceError } from "../Errors.ts"; -import type { ProviderAdapterCapabilities, ProviderAdapterShape } from "./ProviderAdapter.ts"; +import type { + ProviderAdapterCapabilities, + ProviderAdapterShape, + ProviderStopConfirmation, +} from "./ProviderAdapter.ts"; import type { ProviderInstanceRoutingInfo } from "./ProviderAdapterRegistry.ts"; /** @@ -307,6 +311,17 @@ export interface ProviderServiceShape { */ readonly listSessions: () => Effect.Effect>; + /** + * Ends every runtime of a thread and proves it gone, before the thread + * moves to another computer (docs/design/fleet.md, Part 3b, "The + * barrier"): each session whose key belongs to the thread (its own agent, + * room agents, side answers, early starts) is stopped, then every driver + * confirms its processes, and what they started, exited. `unconfirmed` + * when any can't be proven, with why. It doesn't fence the thread: the + * caller must already have stopped new sessions (ThreadSessionAdmission). + */ + readonly quiesceThread: (threadId: ThreadId) => Effect.Effect; + /** * Read capabilities for the adapter bound to a configured provider instance. */ diff --git a/apps/server/src/provider/Services/ThreadSessionAdmission.ts b/apps/server/src/provider/Services/ThreadSessionAdmission.ts new file mode 100644 index 000000000..d191ea008 --- /dev/null +++ b/apps/server/src/provider/Services/ThreadSessionAdmission.ts @@ -0,0 +1,33 @@ +/** + * ThreadSessionAdmission - whether a thread's agent sessions may start. + * + * ProviderService asks it before it starts or recovers any session, whatever + * its key (the thread's own agent, a room agent, a side answer, an early + * start) and whoever asked (a turn, a transcript read, compaction, startup). + * A command guard can't stop a session that reopens without a command; this + * can. It refuses while a thread's queued move is ending its sessions + * (`pendingMove.stage: "quiescing"`) and once the thread moves or moved + * (docs/design/fleet.md, Part 3b, "The session fence"). + * + * @module ThreadSessionAdmission + */ +import type { ThreadId } from "@threadlines/contracts"; +import * as Context from "effect/Context"; +import * as Effect from "effect/Effect"; +import * as Layer from "effect/Layer"; + +export interface ThreadSessionAdmissionShape { + /** Why no session of `threadId` (a thread id, not a session key) may start now, or null. */ + readonly refusal: (threadId: ThreadId) => Effect.Effect; +} + +export class ThreadSessionAdmission extends Context.Service< + ThreadSessionAdmission, + ThreadSessionAdmissionShape +>()("threadlines/provider/Services/ThreadSessionAdmission") {} + +/** Admits every session: for tests of code that never moves a thread. */ +export const ThreadSessionAdmissionPermissive = Layer.succeed( + ThreadSessionAdmission, + ThreadSessionAdmission.of({ refusal: () => Effect.succeed(null) }), +); diff --git a/apps/server/src/provider/agentProcessLedger.ts b/apps/server/src/provider/agentProcessLedger.ts new file mode 100644 index 000000000..768e6ed85 --- /dev/null +++ b/apps/server/src/provider/agentProcessLedger.ts @@ -0,0 +1,241 @@ +// @effect-diagnostics nodeBuiltinImport:off +/** + * Proof that an agent's processes are gone (docs/design/fleet.md, Part 3b, + * "Confirmed teardown"). A driver tracks each agent process it starts here, + * by session key; its stop path calls `beforeStop` while the process still + * lives, which notes every process it started (in any process group, so a + * shell an agent put in a group of its own is found too); and + * `confirmThreadStopped` asks `confirm`, which waits for the agent to exit, + * ends what outlived it, and says whether everything is gone. + * + * On POSIX each tracked process leads its own process group (Effect's + * spawner detaches by default, and the Claude driver spawns its CLI that + * way), so the group is checked as a whole. Windows has no process groups + * and no cheap process table: there the proof is that the agent's process + * exited after its whole tree was killed (`taskkill /T`), which can miss a + * process that left the tree. + */ +import { spawnSync } from "node:child_process"; + +import { listDescendantProcessIds } from "@threadlines/shared/processTree"; +import { parseSessionKey } from "@threadlines/shared/threadParticipants"; +import type { ThreadId } from "@threadlines/contracts"; +import * as Duration from "effect/Duration"; +import * as Effect from "effect/Effect"; +import * as Option from "effect/Option"; +import type * as Scope from "effect/Scope"; + +import type { ProviderStopConfirmation } from "./Services/ProviderAdapter.ts"; + +/** An agent process a driver started. */ +export interface TrackedAgentProcess { + readonly pid: number; + /** Succeeds once the process exited (at once if it already has). */ + readonly exited: Effect.Effect; + /** + * It leads a process group of its own (POSIX, spawned detached), so the + * whole group is its. Default: true on POSIX. + */ + readonly ownGroup?: boolean; +} + +/** How the ledger looks at and signals processes; real by default, swapped in tests. */ +export interface ProcessProbe { + readonly platform: NodeJS.Platform; + /** The process (or, with a negative pid, any process of the group) still exists. */ + readonly exists: (pid: number) => boolean; + readonly signal: (pid: number, signal: NodeJS.Signals) => void; + /** Every live descendant, from one snapshot of the process table. */ + readonly descendants: (pid: number) => ReadonlyArray; + /** Windows: kills the process and its tree. */ + readonly killTree: (pid: number) => void; +} + +export const realProcessProbe: ProcessProbe = { + platform: process.platform, + exists: (pid) => { + try { + process.kill(pid, 0); + return true; + } catch (cause) { + // Not ours to signal, but there. + return (cause as NodeJS.ErrnoException).code === "EPERM"; + } + }, + signal: (pid, signal) => { + try { + process.kill(pid, signal); + } catch { + // Already gone. + } + }, + descendants: listDescendantProcessIds, + killTree: (pid) => { + spawnSync("taskkill", ["/pid", String(pid), "/T", "/F"], { + stdio: "ignore", + windowsHide: true, + }); + }, +}; + +export interface AgentProcessLedger { + /** A process started for `sessionKey`. Forgotten once it and all it started are gone. */ + readonly track: (sessionKey: ThreadId, agent: TrackedAgentProcess) => Effect.Effect; + /** + * The driver is about to stop `sessionKey`'s runtime: note what its + * processes started, while they can still be told apart from anyone + * else's (once a process dies its children belong to pid 1). + */ + readonly beforeStop: (sessionKey: ThreadId) => Effect.Effect; + /** Everything tracked for `threadId`'s session keys is gone, or why that can't be said. */ + readonly confirm: (threadId: ThreadId) => Effect.Effect; +} + +export interface AgentProcessLedgerOptions { + readonly probe?: ProcessProbe; + /** How long `confirm` waits for an agent to exit before killing it. */ + readonly exitWait?: Duration.Input; + /** How long it waits for what it killed to be gone. */ + readonly settleWait?: Duration.Input; + /** + * How long after `beforeStop` a noted process may still be signalled. A + * pid noted earlier than this could belong to someone else by now: it is + * only reported, never signalled. + */ + readonly trustWindow?: Duration.Input; +} + +interface Entry { + readonly sessionKey: ThreadId; + readonly pid: number; + readonly exited: Effect.Effect; + readonly ownGroup: boolean; + leaderGone: boolean; + /** What it had started when its stop began, and when that was noted. */ + noted: ReadonlyArray; + notedAt: number; +} + +const POLL = Duration.millis(50); + +export const makeAgentProcessLedger = ( + options?: AgentProcessLedgerOptions, +): Effect.Effect => + Effect.gen(function* () { + const probe = options?.probe ?? realProcessProbe; + const posix = probe.platform !== "win32"; + const exitWait = Duration.fromInputUnsafe(options?.exitWait ?? Duration.seconds(10)); + const settleWait = Duration.fromInputUnsafe(options?.settleWait ?? Duration.seconds(3)); + const trustWindowMs = Duration.toMillis( + Duration.fromInputUnsafe(options?.trustWindow ?? Duration.minutes(2)), + ); + const scope = yield* Effect.scope; + const entries = new Set(); + + const groupAlive = (entry: Entry) => + posix && entry.ownGroup + ? probe.exists(-entry.pid) + : !entry.leaderGone && probe.exists(entry.pid); + const notedAlive = (entry: Entry) => entry.noted.filter((pid) => probe.exists(pid)); + const settled = (entry: Entry) => + entry.leaderGone && !groupAlive(entry) && notedAlive(entry).length === 0; + + const track: AgentProcessLedger["track"] = (sessionKey, agent) => + Effect.gen(function* () { + const entry: Entry = { + sessionKey, + pid: agent.pid, + exited: agent.exited, + ownGroup: agent.ownGroup ?? posix, + leaderGone: false, + noted: [], + notedAt: 0, + }; + entries.add(entry); + // A process that ends with nothing left behind needs no proof later. + yield* agent.exited.pipe( + Effect.andThen( + Effect.sync(() => { + entry.leaderGone = true; + if (settled(entry)) entries.delete(entry); + }), + ), + Effect.forkIn(scope), + ); + }); + + const beforeStop: AgentProcessLedger["beforeStop"] = (sessionKey) => + Effect.sync(() => { + for (const entry of entries) { + if (entry.sessionKey !== sessionKey || entry.leaderGone) continue; + entry.noted = probe.descendants(entry.pid); + entry.notedAt = Date.now(); + } + }); + + /** Waits for one tracked process and what it left; the problems that remain. */ + const settle = (entry: Entry) => + Effect.gen(function* () { + const problems: string[] = []; + if (Option.isNone(yield* entry.exited.pipe(Effect.timeoutOption(exitWait)))) { + if (posix && entry.ownGroup) probe.signal(-entry.pid, "SIGKILL"); + else if (posix) probe.signal(entry.pid, "SIGKILL"); + else probe.killTree(entry.pid); + if (Option.isNone(yield* entry.exited.pipe(Effect.timeoutOption(settleWait)))) { + problems.push(`its process ${entry.pid} didn't exit`); + return problems; + } + } + entry.leaderGone = true; + const trusted = Date.now() - entry.notedAt <= trustWindowMs; + if (groupAlive(entry)) probe.signal(-entry.pid, "SIGKILL"); + if (trusted) { + for (const pid of notedAlive(entry)) { + probe.signal(pid, "SIGKILL"); + // A shell that made a group of its own leads it. + if (posix) probe.signal(-pid, "SIGKILL"); + } + } + const deadline = Date.now() + Duration.toMillis(settleWait); + while (!settled(entry) && Date.now() < deadline) { + yield* Effect.sleep(POLL); + } + if (groupAlive(entry)) { + problems.push(`processes in its process group ${entry.pid} are still running`); + } + const left = notedAlive(entry); + if (left.length > 0) { + problems.push( + trusted + ? `${left.length} process${left.length === 1 ? "" : "es"} it started (${left.join(", ")}) are still running` + : `${left.length} process${left.length === 1 ? "" : "es"} it started earlier (${left.join(", ")}) may still be running`, + ); + } + return problems; + }); + + const confirm: AgentProcessLedger["confirm"] = (threadId) => + Effect.gen(function* () { + const mine = [...entries].filter( + (entry) => parseSessionKey(entry.sessionKey).threadId === threadId, + ); + const problems = yield* Effect.forEach( + mine, + (entry) => + settle(entry).pipe( + Effect.tap((left) => + Effect.sync(() => { + if (left.length === 0) entries.delete(entry); + }), + ), + ), + { concurrency: "unbounded" }, + ); + const all = problems.flat(); + return all.length === 0 + ? ({ confirmed: true } as const) + : ({ confirmed: false, reason: all.join("; ") } as const); + }); + + return { track, beforeStop, confirm } satisfies AgentProcessLedger; + }); diff --git a/apps/server/src/server.ts b/apps/server/src/server.ts index 2761193b2..ccc2e1eba 100644 --- a/apps/server/src/server.ts +++ b/apps/server/src/server.ts @@ -59,6 +59,7 @@ import { ProviderSessionRuntimeRepositoryLive } from "./persistence/Layers/Provi import { ProviderAdapterRegistryLive } from "./provider/Layers/ProviderAdapterRegistry.ts"; import { ProviderEventLoggersLive } from "./provider/Layers/ProviderEventLoggers.ts"; import { ProviderServiceLive } from "./provider/Layers/ProviderService.ts"; +import { ThreadSessionAdmissionLive } from "./provider/Layers/ThreadSessionAdmission.ts"; import { ProviderSessionReaperLive } from "./provider/Layers/ProviderSessionReaper.ts"; import { ThreadAutoArchiveSweeperLive } from "./orchestration/Layers/ThreadAutoArchiveSweeper.ts"; import { ContinueAtResetSweeperLive } from "./orchestration/Layers/ContinueAtResetSweeper.ts"; @@ -398,6 +399,8 @@ const ProviderSessionDirectoryLayerLive = ProviderSessionDirectoryLive.pipe( const ProviderLayerLive = ProviderServiceLive.pipe( Layer.provide(ProviderAdapterRegistryLive), Layer.provideMerge(ProviderSessionDirectoryLayerLive), + // The session fence reads the thread's row (docs/design/fleet.md, Part 3b). + Layer.provide(ThreadSessionAdmissionLive.pipe(Layer.provide(ProjectionThreadRepositoryLive))), ); const PersistenceLayerLive = Layer.empty.pipe(Layer.provideMerge(SqlitePersistenceLayerLive)); From f68120900d5f1c9f5d6a42f939a0f961561598a8 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:01:46 -0500 Subject: [PATCH 05/48] feat(contracts): a move target that can't be reached says so ThreadMoveTargetPreview's refusalCode gains unreachable, so a client can still queue a move to a computer that is offline right now. --- packages/contracts/src/moves.ts | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/packages/contracts/src/moves.ts b/packages/contracts/src/moves.ts index 4c1a91eb7..8f79ece68 100644 --- a/packages/contracts/src/moves.ts +++ b/packages/contracts/src/moves.ts @@ -24,8 +24,12 @@ export const ThreadMoveTargetPreview = Schema.Struct({ label: TrimmedNonEmptyString, /** Why it can't take this thread, in words for the user. Absent: it can. */ refusal: Schema.optional(TrimmedNonEmptyString), - /** `no_project`: it has no copy of this project yet; the app can offer to set one up. */ - refusalCode: Schema.optional(Schema.Literals(["no_project"])), + /** + * `no_project`: it has no copy of this project yet; the app can offer to + * set one up. `unreachable`: it can't be reached right now; a queued move + * (`moves.queue`) may still pick it, and waits for it to connect. + */ + refusalCode: Schema.optional(Schema.Literals(["no_project", "unreachable"])), /** Its project the thread would land in. */ projectName: Schema.optional(TrimmedNonEmptyString), /** Whether its user is asked first, or it takes the move at once. */ From ed0b78ed3dd4ee22256a3fbf76e24bb2f0d014b0 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:01:46 -0500 Subject: [PATCH 06/48] feat(server): queued moves start once their thread is free MoveQueue (moves/MoveQueue.ts) queues a move after ThreadMoves checks the link, cancels it by request (or cancels the transfer when the decider already started it), and runs a reactor: it subscribes before its startup scan, starts after the providers settled, and watches thread and link changes. A block ends the wait with why; a wait is reported; a free thread goes through the barrier (quiesce, a proven stop, a fresh look) to ThreadMoves.startQueued. An unreachable target or an unproven stop holds the move and tries again from 15 seconds up to 2 minutes, at once when a link changes. v1's export waits for a proven stop too, the dialog's preview says what the thread waits for, the moves RPCs get their own layer with queue, unqueue and retry, and early agent starts skip a thread whose move waits. --- apps/server/src/links/ComputerLinkReactors.ts | 4 + apps/server/src/moves/MoveQueue.ts | 538 ++++++++++++++++++ apps/server/src/moves/ThreadMoves.ts | 196 ++++++- .../Layers/ProviderCommandReactor.ts | 8 +- .../src/orchestration/threadMoveDecisions.ts | 7 +- apps/server/src/server.test.ts | 2 + apps/server/src/server.ts | 6 +- apps/server/src/ws.ts | 46 +- 8 files changed, 761 insertions(+), 46 deletions(-) create mode 100644 apps/server/src/moves/MoveQueue.ts diff --git a/apps/server/src/links/ComputerLinkReactors.ts b/apps/server/src/links/ComputerLinkReactors.ts index bb9ee4b6c..439cecc88 100644 --- a/apps/server/src/links/ComputerLinkReactors.ts +++ b/apps/server/src/links/ComputerLinkReactors.ts @@ -11,6 +11,7 @@ import * as Effect from "effect/Effect"; import * as Layer from "effect/Layer"; import type * as Scope from "effect/Scope"; +import { MoveQueue } from "../moves/MoveQueue.ts"; import { ThreadMoves } from "../moves/ThreadMoves.ts"; import { LinkMoves } from "./LinkMoves.ts"; import { LinkRequests } from "./LinkRequests.ts"; @@ -32,12 +33,15 @@ export const ComputerLinkReactorsLive = Layer.effect( const linkRequests = yield* LinkRequests; const linkMoves = yield* LinkMoves; const threadMoves = yield* ThreadMoves; + const moveQueue = yield* MoveQueue; return ComputerLinkReactors.of({ start: () => Effect.gen(function* () { yield* linkRequests.startReactor(); yield* linkMoves.startReactor(); yield* threadMoves.startReactor(); + // Moves that wait for their thread (docs/design/fleet.md, Part 3b). + yield* moveQueue.startReactor(); yield* remoteChildren.start(); }), }); diff --git a/apps/server/src/moves/MoveQueue.ts b/apps/server/src/moves/MoveQueue.ts new file mode 100644 index 000000000..2f94abb02 --- /dev/null +++ b/apps/server/src/moves/MoveQueue.ts @@ -0,0 +1,538 @@ +// @effect-diagnostics nodeBuiltinImport:off +/** + * Moves that wait for their thread to be free (docs/design/fleet.md, Part + * 3b), on the computer the thread leaves. `queue` checks the link and records + * the thread's pending move; `unqueue` cancels it, or the transfer it already + * started; `retry` looks again now at a held one. + * + * The reactor watches threads with a pending move, and the links. For each: + * something only the user can clear ends the wait (`queue-end`, "Didn't + * move: …"); a wait is reported and left alone; a free thread goes through + * the barrier: `thread.move.quiesce` (the session fence goes up and its + * sessions stop), `ProviderService.quiesceThread` (every driver proves its + * agent's processes are gone), a fresh look (a provider turn that slipped + * in sends it back to waiting), and then `ThreadMoves.startQueued`, whose + * `thread.move.start` ends the pending move in the same decision. An + * unreachable target, or a stop that can't be proven, holds the move and + * tries again, from 15 seconds backing off to 2 minutes; a link that + * changes or comes back tries at once. + */ +import { randomUUID } from "node:crypto"; + +import { + CommandId, + type OrchestrationCommand, + type OrchestrationEvent, + type OrchestrationReadModel, + type OrchestrationThread, + type ThreadId, + ThreadMoveError, + type ThreadMoveHold, + ThreadMoveRequestId, + type ThreadMoveQueueInput, + type ThreadMoveQueueResult, + type ThreadMoveRequester, + type ThreadMoveRetryInput, + type ThreadMoveUnqueueInput, + type ThreadMoveWait, +} from "@threadlines/contracts"; +import * as Context from "effect/Context"; +import * as DateTime from "effect/DateTime"; +import * as Duration from "effect/Duration"; +import * as Effect from "effect/Effect"; +import * as Fiber from "effect/Fiber"; +import * as Layer from "effect/Layer"; +import * as Option from "effect/Option"; +import * as Scope from "effect/Scope"; +import * as Stream from "effect/Stream"; + +import { ComputerLinks } from "../links/ComputerLinks.ts"; +import { OrchestrationEngineService } from "../orchestration/Services/OrchestrationEngine.ts"; +import { ProjectionSnapshotQuery } from "../orchestration/Services/ProjectionSnapshotQuery.ts"; +import { + firstMoveBlock, + moveWaitsOf, + type ThreadMoveRefusalEntry, + threadMoveRefusals, +} from "../orchestration/threadMoveDecisions.ts"; +import type { ProviderStopConfirmation } from "../provider/Services/ProviderAdapter.ts"; +import { ProviderService } from "../provider/Services/ProviderService.ts"; +import { type QueuedMoveStart, ThreadMoves, type ThreadMovesShape } from "./ThreadMoves.ts"; + +export interface MoveQueueShape { + readonly queue: ( + input: ThreadMoveQueueInput & { readonly requestedBy?: ThreadMoveRequester }, + ) => Effect.Effect; + readonly unqueue: (input: ThreadMoveUnqueueInput) => Effect.Effect; + readonly retry: (input: ThreadMoveRetryInput) => Effect.Effect; + /** Watch queued moves: the ones the last process left, and every one from now on. */ + readonly startReactor: () => Effect.Effect; +} + +export class MoveQueue extends Context.Service()( + "threadlines/moves/MoveQueue", +) {} + +/** What the move queue needs, narrowed so tests can run it on the real decider. */ +export interface MoveQueueDeps { + readonly dispatch: ( + command: OrchestrationCommand, + ) => Effect.Effect; + /** Subscribed when run, so nothing published after is missed. */ + readonly subscribeEvents: Effect.Effect, never, Scope.Scope>; + /** Something about the links changed: one added, removed, its access, or its connection. */ + readonly linkChanges: Stream.Stream; + /** The thread, archived ones too; null when it's gone. A read that fails is a defect: the look is tried again. */ + readonly readThread: (threadId: ThreadId) => Effect.Effect; + readonly readModel: Effect.Effect; + readonly quiesce: (threadId: ThreadId) => Effect.Effect; + readonly checkQueue: ThreadMovesShape["checkQueue"]; + readonly startQueued: ThreadMovesShape["startQueued"]; + /** v1's cancel of the thread's move under way. */ + readonly cancelMove: ThreadMovesShape["cancel"]; + /** The wait before trying a held move again, by how many times it was held. */ + readonly retryDelay?: (attempt: number) => Duration.Duration; + /** How long to gather a burst of changes before looking. */ + readonly settleDelay?: Duration.Duration; +} + +const nowIso = Effect.map(DateTime.now, DateTime.formatIso); +const moveError = (detail: string) => new ThreadMoveError({ detail }); + +/** 15 s, 30 s, 1 min, then every 2 min. */ +const defaultRetryDelay = (attempt: number) => + Duration.seconds(Math.min(15 * 2 ** Math.max(0, attempt), 120)); + +/** How long after a quiesce it is asked again for a session the projection still shows. */ +const REQUIESCE_AFTER_MS = 10_000; + +/** A thread's move checks, from the thread alone: no other thread or project. */ +const localRefusals = (thread: OrchestrationThread) => + threadMoveRefusals({ projects: [], threads: [thread] }, thread); + +interface TrackedMove { + requestId: ThreadMoveRequestId; + running: boolean; + again: boolean; + timer: Fiber.Fiber | null; + timerDueMs: number; + /** How many times in a row it was held (unreachable, unconfirmed). */ + holds: number; + quiescedAtMs: number; +} + +export const makeMoveQueue = ( + deps: MoveQueueDeps, +): Effect.Effect => + Effect.gen(function* () { + const scope = yield* Effect.scope; + const retryDelay = deps.retryDelay ?? defaultRetryDelay; + const settleDelay = deps.settleDelay ?? Duration.millis(250); + const tracked = new Map(); + + const dispatchIgnoringRefusal = (command: OrchestrationCommand) => + deps.dispatch(command).pipe( + Effect.asVoid, + Effect.catch((error) => + Effect.logInfo("move queue: a command was refused", { + type: command.type, + detail: error.message, + }), + ), + ); + + const commandId = (requestId: string, step: string) => + CommandId.make(`server:thread-move-queue:${requestId}:${step}:${randomUUID()}`); + + const report = ( + threadId: ThreadId, + requestId: ThreadMoveRequestId, + waitsFor: ReadonlyArray, + hold: ThreadMoveHold | null, + ) => + Effect.gen(function* () { + yield* dispatchIgnoringRefusal({ + type: "thread.move.queue-status", + commandId: commandId(requestId, "status"), + threadId, + requestId, + waitsFor: [...waitsFor], + hold, + createdAt: yield* nowIso, + }); + }); + + const end = ( + threadId: ThreadId, + requestId: ThreadMoveRequestId, + reason: "blocked" | "gone", + words: string, + ) => + Effect.gen(function* () { + yield* dispatchIgnoringRefusal({ + type: "thread.move.queue-end", + commandId: commandId(requestId, "end"), + threadId, + requestId, + reason, + words, + createdAt: yield* nowIso, + }); + }); + + // ------------------------------------------------------------------------- + // Scheduling: one look per thread at a time; changes while it looks ask + // for another; a held move has one timer. + // ------------------------------------------------------------------------- + + const schedule = (threadId: ThreadId, delay: Duration.Duration): Effect.Effect => + Effect.gen(function* () { + const entry = tracked.get(threadId); + if (entry === undefined) return; + const dueMs = Date.now() + Duration.toMillis(delay); + if (entry.timer !== null && entry.timerDueMs <= dueMs) return; + if (entry.timer !== null) yield* Fiber.interrupt(entry.timer); + entry.timerDueMs = dueMs; + entry.timer = yield* Effect.sleep(delay).pipe( + Effect.andThen( + Effect.suspend(() => { + entry.timer = null; + return run(threadId); + }), + ), + Effect.forkIn(scope), + ); + }); + + const run = (threadId: ThreadId): Effect.Effect => + Effect.suspend(() => { + const entry = tracked.get(threadId); + if (entry === undefined) return Effect.void; + if (entry.running) { + entry.again = true; + return Effect.void; + } + entry.running = true; + const once = evaluate(threadId, entry).pipe( + Effect.catchCause((cause) => + Effect.logWarning("move queue: looking at a queued move failed", { + threadId, + cause, + }).pipe(Effect.andThen(schedule(threadId, retryDelay(entry.holds)))), + ), + ); + const loop: Effect.Effect = Effect.suspend(() => { + entry.again = false; + return once.pipe( + Effect.andThen( + Effect.suspend(() => + entry.again && tracked.get(threadId) === entry ? loop : Effect.void, + ), + ), + ); + }); + return loop.pipe( + Effect.ensuring( + Effect.sync(() => { + entry.running = false; + }), + ), + ); + }); + + /** Watches a thread's queued move `requestId`, replacing an older one. */ + const track = (threadId: ThreadId, requestId: ThreadMoveRequestId) => + Effect.gen(function* () { + const existing = tracked.get(threadId); + if (existing !== undefined) { + if (existing.requestId === requestId) return; + existing.requestId = requestId; + existing.holds = 0; + existing.quiescedAtMs = 0; + return; + } + tracked.set(threadId, { + requestId, + running: false, + again: false, + timer: null, + timerDueMs: 0, + holds: 0, + quiescedAtMs: 0, + }); + }); + + const forget = (threadId: ThreadId) => + Effect.gen(function* () { + const entry = tracked.get(threadId); + if (entry === undefined) return; + tracked.delete(threadId); + if (entry.timer !== null) yield* Fiber.interrupt(entry.timer); + }); + + const held = (threadId: ThreadId, entry: TrackedMove) => + Effect.suspend(() => { + const delay = retryDelay(entry.holds); + entry.holds += 1; + return schedule(threadId, delay); + }); + + // ------------------------------------------------------------------------- + // One look at one queued move. + // ------------------------------------------------------------------------- + + /** Reports what it waits for, or ends it when something only the user can clear stands in the way. */ + const settleRefusals = ( + threadId: ThreadId, + requestId: ThreadMoveRequestId, + refusals: ReadonlyArray, + ) => + Effect.gen(function* () { + const block = firstMoveBlock(refusals); + if (block !== null) { + const gone = block.code === "archived" || block.code === "deleted"; + yield* end(threadId, requestId, gone ? "gone" : "blocked", block.words); + return true; + } + if (refusals.length > 0) { + yield* report(threadId, requestId, moveWaitsOf(refusals), null); + return true; + } + return false; + }); + + const evaluate = (threadId: ThreadId, entry: TrackedMove) => + Effect.gen(function* () { + const thread = yield* deps.readThread(threadId); + const pending = thread?.pendingMove ?? null; + if (thread === null || pending === null) return yield* forget(threadId); + // A newer queued move than the one this was watching: watch it instead. + yield* track(threadId, pending.requestId); + const requestId = pending.requestId; + // From the thread alone first: while it works, nothing else needs reading. + if (yield* settleRefusals(threadId, requestId, localRefusals(thread))) return; + const model = yield* deps.readModel; + const current = model.threads.find((candidate) => candidate.id === threadId) ?? thread; + if (yield* settleRefusals(threadId, requestId, threadMoveRefusals(model, current))) return; + if ((pending.waitsFor ?? []).length > 0) { + yield* report(threadId, requestId, [], pending.hold ?? null); + } + + // The barrier: the fence goes up and its sessions stop. + if (pending.stage === "waiting" || !sessionStopped(thread)) { + if (pending.stage === "waiting" || Date.now() - entry.quiescedAtMs > REQUIESCE_AFTER_MS) { + entry.quiescedAtMs = Date.now(); + yield* dispatchIgnoringRefusal({ + type: "thread.move.quiesce", + commandId: commandId(requestId, "quiesce"), + threadId, + requestId, + createdAt: yield* nowIso, + }); + } + } + // Every driver proves its agent's processes are gone. + const confirmation = yield* deps.quiesce(threadId); + if (!confirmation.confirmed) { + yield* report(threadId, requestId, [], { + kind: "stop_unconfirmed", + words: `Couldn't confirm the agent stopped: ${confirmation.reason}.`, + }); + return yield* held(threadId, entry); + } + // A turn that slipped in before the fence sends it back to waiting. + const after = yield* deps.readThread(threadId); + if (after === null || after.pendingMove?.requestId !== requestId) { + entry.again = true; + return; + } + if (yield* settleRefusals(threadId, requestId, localRefusals(after))) return; + if (after.pendingMove.stage !== "quiescing" || !sessionStopped(after)) { + // The stop is still landing in the projection; its event looks again. + return yield* schedule(threadId, retryDelay(0)); + } + const outcome: QueuedMoveStart = yield* deps.startQueued({ + threadId, + requestId, + targetEnvironmentId: pending.target.environmentId, + agent: pending.agent, + }); + switch (outcome.kind) { + case "started": + yield* forget(threadId); + return; + case "unreachable": + yield* report(threadId, requestId, [], { + kind: "target_unreachable", + words: outcome.words, + }); + return yield* held(threadId, entry); + case "blocked": + yield* end(threadId, requestId, "blocked", outcome.words); + return; + case "busy": + return yield* schedule(threadId, retryDelay(0)); + } + }); + + // ------------------------------------------------------------------------- + // The service. + // ------------------------------------------------------------------------- + + const queue: MoveQueueShape["queue"] = (input) => + Effect.gen(function* () { + const { label } = yield* deps.checkQueue({ + threadId: input.threadId, + targetEnvironmentId: input.targetEnvironmentId, + agent: input.agent, + }); + const requestId = ThreadMoveRequestId.make(randomUUID()); + yield* deps + .dispatch({ + type: "thread.move.queue", + commandId: CommandId.make(`server:thread-move-queue:${requestId}:queue`), + threadId: input.threadId, + requestId, + target: { environmentId: input.targetEnvironmentId, label }, + agent: input.agent, + mode: input.mode, + requestedBy: input.requestedBy ?? { kind: "user" }, + createdAt: yield* nowIso, + }) + .pipe(Effect.mapError((error) => moveError(error.message))); + return { requestId }; + }); + + const unqueue: MoveQueueShape["unqueue"] = (input) => + Effect.gen(function* () { + const read = deps.readThread(input.threadId); + const thread = yield* read; + if (thread === null) return; + if (thread.pendingMove?.requestId === input.requestId) { + yield* deps + .dispatch({ + type: "thread.move.unqueue", + commandId: commandId(input.requestId, "unqueue"), + threadId: input.threadId, + requestId: input.requestId, + createdAt: yield* nowIso, + }) + .pipe(Effect.mapError((error) => moveError(error.message))); + } + // Read after the cancel: a start the decider took first is now the + // thread's move, and the Cancel cancels its transfer instead. + const move = (yield* read)?.move ?? null; + if (move === null || move.queuedRequestId !== input.requestId) return; + if (move.movedAway !== undefined) { + return yield* moveError(`This thread already moved to ${move.target.label}.`); + } + if (!move.cancelRequested) yield* deps.cancelMove(input.threadId); + }); + + const retry: MoveQueueShape["retry"] = (input) => + Effect.gen(function* () { + const entry = tracked.get(input.threadId); + if (entry === undefined || entry.requestId !== input.requestId) { + return yield* moveError("That move isn't waiting anymore."); + } + entry.holds = 0; + yield* schedule(input.threadId, Duration.zero); + }); + + const startReactor: MoveQueueShape["startReactor"] = () => + Effect.gen(function* () { + // Subscribed before the scan, so a change during it isn't missed. + const events = yield* deps.subscribeEvents; + // The moves the last process left, read off the startup path. + yield* Effect.gen(function* () { + const model = yield* deps.readModel; + for (const thread of model.threads) { + const pending = thread.pendingMove ?? null; + if (pending !== null) { + yield* track(thread.id, pending.requestId); + yield* schedule(thread.id, Duration.zero); + } + } + }).pipe( + Effect.catchCause((cause) => + Effect.logWarning("move queue: the queued moves couldn't be read", { cause }), + ), + Effect.forkScoped, + ); + const lookAtAll = (delay: Duration.Duration, fresh: boolean) => + Effect.forEach( + [...tracked.entries()], + ([threadId, entry]) => { + if (fresh) entry.holds = 0; + return schedule(threadId, delay); + }, + { discard: true }, + ); + yield* events.pipe( + Stream.runForEach((event) => + Effect.gen(function* () { + if (event.type === "thread.move-queued") { + yield* track(event.payload.threadId, event.payload.pendingMove.requestId); + return yield* schedule(event.payload.threadId, Duration.zero); + } + if (event.type === "thread.move-queue-ended") { + const entry = tracked.get(event.payload.threadId); + if (entry?.requestId === event.payload.requestId) { + yield* forget(event.payload.threadId); + } + return; + } + // Another thread working in the same folder holds a move: any + // session change anywhere is a reason to look again. + if (event.type === "thread.session-set") { + return yield* lookAtAll(settleDelay, false); + } + const threadId = "threadId" in event.payload ? event.payload.threadId : undefined; + if (threadId !== undefined && tracked.has(threadId)) { + yield* schedule(threadId, settleDelay); + } + }), + ), + Effect.forkScoped, + ); + // A link added, removed, changed or back: held moves try again at once. + yield* deps.linkChanges.pipe( + Stream.runForEach(() => lookAtAll(settleDelay, true)), + Effect.catchCause((cause) => + Effect.logWarning("move queue: link changes stopped", { cause }), + ), + Effect.forkScoped, + ); + }); + + return MoveQueue.of({ queue, unqueue, retry, startReactor }); + }); + +/** Whether the thread's agent has no session left, as projected. */ +const sessionStopped = (thread: OrchestrationThread) => + thread.session === null || thread.session.status === "stopped"; + +export const MoveQueueLive = Layer.effect( + MoveQueue, + Effect.gen(function* () { + const engine = yield* OrchestrationEngineService; + const snapshots = yield* ProjectionSnapshotQuery; + const providers = yield* ProviderService; + const threadMoves = yield* ThreadMoves; + const links = yield* ComputerLinks; + return yield* makeMoveQueue({ + dispatch: engine.dispatch, + subscribeEvents: engine.subscribeDomainEvents, + linkChanges: links.changes.pipe(Stream.orDie), + readThread: (threadId) => + snapshots + .getThreadDetailById(threadId, { includeArchived: true }) + .pipe(Effect.map(Option.getOrNull), Effect.orDie), + readModel: snapshots.getCommandReadModel().pipe(Effect.orDie), + quiesce: providers.quiesceThread, + checkQueue: threadMoves.checkQueue, + startQueued: threadMoves.startQueued, + cancelMove: threadMoves.cancel, + }); + }), +); diff --git a/apps/server/src/moves/ThreadMoves.ts b/apps/server/src/moves/ThreadMoves.ts index 28a6890b6..75edbe4bf 100644 --- a/apps/server/src/moves/ThreadMoves.ts +++ b/apps/server/src/moves/ThreadMoves.ts @@ -28,6 +28,7 @@ import { type ThreadId, ThreadMoveError, ThreadMoveId, + type ThreadMoveRequestId, type ThreadMovePhase, type ThreadMovePreview, type ThreadMoveStartInput, @@ -51,7 +52,13 @@ import { ServerConfig } from "../config.ts"; import { ServerEnvironment } from "../environment/Services/ServerEnvironment.ts"; import { ComputerLinkClient, type PeerClient } from "../links/ComputerLinkClient.ts"; import { ComputerLinkRepository, isUsableLink } from "../links/ComputerLinkRepository.ts"; -import { threadMoveRefusal } from "../orchestration/threadMoveDecisions.ts"; +import { + firstMoveBlock, + moveWaitsOf, + threadMoveRefusal, + threadMoveRefusals, +} from "../orchestration/threadMoveDecisions.ts"; +import { ProviderService } from "../provider/Services/ProviderService.ts"; import { OrchestrationEngineService } from "../orchestration/Services/OrchestrationEngine.ts"; import { ProjectionSnapshotQuery } from "../orchestration/Services/ProjectionSnapshotQuery.ts"; import { ProjectionThreadRepository } from "../persistence/Services/ProjectionThreads.ts"; @@ -68,11 +75,42 @@ import { writeMoveExport, } from "./moveExport.ts"; +/** + * How starting a queued move went (`startQueued`), for the move queue to act + * on: `started`; `unreachable`, the target can't be reached now (try again); + * `blocked`, it won't pass without the user (link removed or off, agent or + * project gone there, or something only the user clears here); `busy`, the + * thread or its session isn't ready yet (look again). + */ +export type QueuedMoveStart = + | { readonly kind: "started"; readonly transferId: ThreadMoveId } + | { readonly kind: "unreachable"; readonly words: string } + | { readonly kind: "blocked"; readonly words: string } + | { readonly kind: "busy"; readonly words: string }; + export interface ThreadMovesShape { readonly preview: (threadId: ThreadId) => Effect.Effect; readonly start: ( input: ThreadMoveStartInput, ) => Effect.Effect<{ readonly transferId: ThreadMoveId }, ThreadMoveError>; + /** + * Checks a move to queue (docs/design/fleet.md, Part 3b): the link, and, + * when the target can be reached, its project and agent. An unreachable + * target is allowed: the queued move waits for it. Returns the target's + * name, or why it can't be queued. + */ + readonly checkQueue: (input: { + readonly threadId: ThreadId; + readonly targetEnvironmentId: EnvironmentId; + readonly agent: ModelSelection; + }) => Effect.Effect<{ readonly label: string }, ThreadMoveError>; + /** Starts the thread's queued move `requestId`, now that the thread is free and quiesced. */ + readonly startQueued: (input: { + readonly threadId: ThreadId; + readonly requestId: ThreadMoveRequestId; + readonly targetEnvironmentId: EnvironmentId; + readonly agent: ModelSelection; + }) => Effect.Effect; readonly cancel: (threadId: ThreadId) => Effect.Effect; /** Run unfinished moves to their end, and every move started from now on. */ readonly startReactor: () => Effect.Effect; @@ -119,6 +157,7 @@ const make = Effect.gen(function* () { const git = yield* GitVcsDriver; const config = yield* ServerConfig; const environment = yield* ServerEnvironment; + const providers = yield* ProviderService; const descriptor = yield* environment.getDescriptor; const moveDir = (transferId: string) => NodePath.join(config.stateDir, "moves", transferId); @@ -159,6 +198,7 @@ const make = Effect.gen(function* () { environmentId, label, refusal: `${label} can't be reached right now.`, + refusalCode: "unreachable" as const, agents: [], }; } @@ -200,13 +240,25 @@ const make = Effect.gen(function* () { }), ); - const preview: ThreadMovesShape["preview"] = (threadId) => + /** + * Where the thread could go and what would go with it. `refusal` is what + * only the user can clear (or a project the other computer can't find); + * `waitsFor` is what the thread is busy with. `forQueued` names the queued + * move being started, which doesn't count against itself. + */ + const inspect = (threadId: ThreadId, forQueued?: ThreadMoveRequestId) => Effect.gen(function* () { const model = yield* readModel; const thread = model.threads.find((entry) => entry.id === threadId); if (thread === undefined) return yield* moveError("That thread doesn't exist."); const project = model.projects.find((entry) => entry.id === thread.projectId); - let refusal = threadMoveRefusal(model, thread); + const refusals = threadMoveRefusals(model, thread); + const pending = thread.pendingMove ?? null; + let refusal: string | null = + firstMoveBlock(refusals)?.words ?? + (pending !== null && pending.requestId !== forQueued + ? `A move to ${pending.target.label} is already waiting. Cancel it first.` + : null); let changes: ThreadMovePreview["changes"]; let place: ProjectPlace | null = null; if (project !== undefined && project.kind !== "general-chat") { @@ -234,41 +286,119 @@ const make = Effect.gen(function* () { (row) => describeTarget(row.remoteEnvironmentId, row.remoteLabel, thread, place), { concurrency: "unbounded" }, ); + const waitsFor = moveWaitsOf(refusals); return { - ...(refusal !== null ? { refusal } : {}), - ...(changes !== undefined ? { changes } : {}), - targets, + model, + thread, + // A move that doesn't wait refuses a busy thread too. + busy: threadMoveRefusal(model, thread), + preview: { + ...(refusal !== null ? { refusal } : {}), + ...(waitsFor.length > 0 ? { waitsFor } : {}), + ...(changes !== undefined ? { changes } : {}), + targets, + } satisfies ThreadMovePreview, }; }); + const preview: ThreadMovesShape["preview"] = (threadId) => + inspect(threadId).pipe(Effect.map((inspected) => inspected.preview)); + + /** The target and agent a move names, checked against what the target said. */ + const checkTarget = ( + shown: ThreadMovePreview, + environmentId: EnvironmentId, + agent: ModelSelection, + ): + | { readonly ok: ThreadMoveTargetPreview } + | { readonly unreachable: ThreadMoveTargetPreview } + | { readonly refusal: string } => { + const target = shown.targets.find((entry) => entry.environmentId === environmentId); + if (target === undefined) return { refusal: "This computer isn't linked to that one." }; + if (target.refusalCode === "unreachable") return { unreachable: target }; + if (target.refusal !== undefined) return { refusal: target.refusal }; + const offered = target.agents.find((entry) => entry.instanceId === agent.instanceId); + if (offered === undefined || !offered.models.some((model) => model.model === agent.model)) { + return { refusal: `${target.label} has no such agent or model.` }; + } + return { ok: target }; + }; + + /** Freezes the thread for its move (`thread.move.start`); the decider checks it again. */ + const dispatchStart = ( + threadId: ThreadId, + target: ThreadMoveTargetPreview, + agent: ModelSelection, + queuedRequestId?: ThreadMoveRequestId, + ) => + Effect.gen(function* () { + const transferId = ThreadMoveId.make(randomUUID()); + yield* engine.dispatch({ + type: "thread.move.start", + commandId: CommandId.make(`server:thread-move:${transferId}:start`), + threadId, + transferId, + target: { environmentId: target.environmentId, label: target.label }, + agent, + ...(queuedRequestId !== undefined ? { queuedRequestId } : {}), + createdAt: yield* nowIso, + }); + return transferId; + }); + const start: ThreadMovesShape["start"] = (input) => Effect.gen(function* () { - const shown = yield* preview(input.threadId); + const inspected = yield* inspect(input.threadId); + const shown = inspected.preview; if (shown.refusal !== undefined) return yield* moveError(shown.refusal); - const target = shown.targets.find( - (entry) => entry.environmentId === input.targetEnvironmentId, - ); - if (target === undefined) return yield* moveError("This computer isn't linked to that one."); - if (target.refusal !== undefined) return yield* moveError(target.refusal); - const agent = target.agents.find((entry) => entry.instanceId === input.agent.instanceId); - if (agent === undefined || !agent.models.some((model) => model.model === input.agent.model)) { - return yield* moveError(`${target.label} has no such agent or model.`); + if (inspected.busy !== null) return yield* moveError(inspected.busy); + const checked = checkTarget(shown, input.targetEnvironmentId, input.agent); + if ("refusal" in checked) return yield* moveError(checked.refusal); + if ("unreachable" in checked) { + return yield* moveError(checked.unreachable.refusal ?? "That computer can't be reached."); } - const transferId = ThreadMoveId.make(randomUUID()); - yield* engine - .dispatch({ - type: "thread.move.start", - commandId: CommandId.make(`server:thread-move:${transferId}:start`), - threadId: input.threadId, - transferId, - target: { environmentId: target.environmentId, label: target.label }, - agent: input.agent, - createdAt: yield* nowIso, - }) - .pipe(Effect.mapError((error) => moveError(error.message))); + const transferId = yield* dispatchStart(input.threadId, checked.ok, input.agent).pipe( + Effect.mapError((error) => moveError(error.message)), + ); return { transferId }; }); + const checkQueue: ThreadMovesShape["checkQueue"] = (input) => + Effect.gen(function* () { + const shown = (yield* inspect(input.threadId)).preview; + if (shown.refusal !== undefined) return yield* moveError(shown.refusal); + const checked = checkTarget(shown, input.targetEnvironmentId, input.agent); + if ("refusal" in checked) return yield* moveError(checked.refusal); + return { label: "ok" in checked ? checked.ok.label : checked.unreachable.label }; + }); + + const startQueued: ThreadMovesShape["startQueued"] = (input) => + Effect.gen(function* () { + const inspected = yield* inspect(input.threadId, input.requestId).pipe(Effect.result); + if (inspected._tag === "Failure") { + return { kind: "busy", words: inspected.failure.detail } as const; + } + const shown = inspected.success.preview; + if (shown.refusal !== undefined) return { kind: "blocked", words: shown.refusal } as const; + const checked = checkTarget(shown, input.targetEnvironmentId, input.agent); + if ("refusal" in checked) return { kind: "blocked", words: checked.refusal } as const; + if ("unreachable" in checked) { + return { + kind: "unreachable", + words: `Waiting for ${checked.unreachable.label} to connect.`, + } as const; + } + const started = yield* dispatchStart( + input.threadId, + checked.ok, + input.agent, + input.requestId, + ).pipe(Effect.result); + return started._tag === "Success" + ? ({ kind: "started", transferId: started.success } as const) + : ({ kind: "busy", words: started.failure.message } as const); + }); + /** * A thread's move, read from its row so an archived thread's move runs * on; none for a deleted thread. Fails when it couldn't be read. @@ -638,6 +768,16 @@ const make = Effect.gen(function* () { } const thread = (yield* readModel).threads.find((entry) => entry.id === threadId); if (thread === undefined) return; + // Nothing of the thread may still run while it's packed: its agent's + // processes must be proven gone (docs/design/fleet.md, Part 3b). Not + // proven, this attempt stops and the worker tries again; the thread + // stays frozen, and Cancel still works. + const quiesced = yield* providers.quiesceThread(threadId); + if (!quiesced.confirmed) { + return yield* new ThreadMoveFailure({ + reason: `Couldn't confirm the agent stopped: ${quiesced.reason}.`, + }); + } const manifest = yield* exportFor(thread, move.agent, transferId, false).pipe( Effect.result, ); @@ -749,7 +889,7 @@ const make = Effect.gen(function* () { ); }); - return ThreadMoves.of({ preview, start, cancel, startReactor }); + return ThreadMoves.of({ preview, start, checkQueue, startQueued, cancel, startReactor }); }); export const ThreadMovesLive = Layer.effect(ThreadMoves, make); diff --git a/apps/server/src/orchestration/Layers/ProviderCommandReactor.ts b/apps/server/src/orchestration/Layers/ProviderCommandReactor.ts index cdad52849..ec0e9e5d2 100644 --- a/apps/server/src/orchestration/Layers/ProviderCommandReactor.ts +++ b/apps/server/src/orchestration/Layers/ProviderCommandReactor.ts @@ -6688,7 +6688,13 @@ const make = Effect.gen(function* () { } const thread = yield* resolveThread(input.threadId); // Moving or moved to another computer: frozen here, nothing to start. - if (!thread || thread.archivedAt !== null || (thread.move ?? null) !== null) { + // A move waiting for the thread takes no new work either. + if ( + !thread || + thread.archivedAt !== null || + (thread.move ?? null) !== null || + (thread.pendingMove ?? null) !== null + ) { return; } // A turn running or on its way: the message will wait or steer anyway. diff --git a/apps/server/src/orchestration/threadMoveDecisions.ts b/apps/server/src/orchestration/threadMoveDecisions.ts index 6cb4c8b2b..949015568 100644 --- a/apps/server/src/orchestration/threadMoveDecisions.ts +++ b/apps/server/src/orchestration/threadMoveDecisions.ts @@ -100,8 +100,11 @@ const blocked = (code: ThreadMoveBlockCode, words: string): ThreadMoveRefusalEnt words, }); +/** What the move checks read: the projects, and the threads they can see. */ +export type ThreadMoveView = Pick; + /** A thread's checkout: its worktree, else its project's folder. */ -function checkoutOf(readModel: OrchestrationReadModel, thread: OrchestrationThread) { +function checkoutOf(readModel: ThreadMoveView, thread: OrchestrationThread) { return ( thread.worktreePath ?? readModel.projects.find((entry) => entry.id === thread.projectId)?.workspaceRoot ?? @@ -118,7 +121,7 @@ function checkoutOf(readModel: OrchestrationReadModel, thread: OrchestrationThre * pending move isn't one of them: each caller decides about that itself. */ export function threadMoveRefusals( - readModel: OrchestrationReadModel, + readModel: ThreadMoveView, thread: OrchestrationThread, ): ReadonlyArray { const found: ThreadMoveRefusalEntry[] = []; diff --git a/apps/server/src/server.test.ts b/apps/server/src/server.test.ts index 9d27286a8..d6898a58f 100644 --- a/apps/server/src/server.test.ts +++ b/apps/server/src/server.test.ts @@ -107,6 +107,7 @@ import { ThemeLibrary, type ThemeLibraryShape } from "./themes/ThemeLibrary.ts"; import { ThemeShowcase } from "./themes/ThemeShowcase.ts"; import { LinkMoves } from "./links/LinkMoves.ts"; import { LinkRequests, type LinkRequestsShape } from "./links/LinkRequests.ts"; +import { MoveQueue } from "./moves/MoveQueue.ts"; import { ThreadMoves } from "./moves/ThreadMoves.ts"; import { ProjectSetups } from "./projectSetup/ProjectSetups.ts"; import { ProjectSetupSource } from "./projectSetup/ProjectSetupSource.ts"; @@ -885,6 +886,7 @@ const buildAppUnderTest = (options?: { pendingChanges: Stream.empty, }), Layer.mock(ThreadMoves)({}), + Layer.mock(MoveQueue)({}), Layer.mock(ProjectSetups)({}), Layer.mock(ProjectSetupSource)({}), Layer.mock(OutsideAgentSignIn)({ bootId: "test-boot" }), diff --git a/apps/server/src/server.ts b/apps/server/src/server.ts index ccc2e1eba..3a1d33d80 100644 --- a/apps/server/src/server.ts +++ b/apps/server/src/server.ts @@ -25,6 +25,7 @@ import { ComputerLinkRepositoryLive } from "./links/ComputerLinkRepository.ts"; import { ComputerLinksLive } from "./links/ComputerLinks.ts"; import { LinkMoveRepositoryLive } from "./links/LinkMoveRepository.ts"; import { LinkMovesLive } from "./links/LinkMoves.ts"; +import { MoveQueueLive } from "./moves/MoveQueue.ts"; import { ThreadMovesLive } from "./moves/ThreadMoves.ts"; import { layer as ProcessRunnerLive } from "./processRunner.ts"; import { ProjectSetupRepositoryLive } from "./projectSetup/ProjectSetupRepository.ts"; @@ -298,7 +299,10 @@ const PlatformServicesLive = ChildEnvironment.layer.pipe( // computers and what it does for computers linked to it. Each side's reactor // starts with the orchestration reactor; runtime ingestion hands turns that // end in a linked computer's thread to the ledger (RemoteRequestTurnEnds). -const ComputerLinksLayerLive = Layer.mergeAll(ComputerLinkReactorsLive, ComputerLinksLive).pipe( +const ComputerLinksLayerLive = ComputerLinkReactorsLive.pipe( + // Moves that wait for their thread watch the links too. + Layer.provideMerge(MoveQueueLive), + Layer.provideMerge(ComputerLinksLive), Layer.provideMerge(RemoteChildReactorLive), Layer.provideMerge(ThreadMovesLive), Layer.provideMerge(LinkMovesLive), diff --git a/apps/server/src/ws.ts b/apps/server/src/ws.ts index a18b82a59..0d73e1fee 100644 --- a/apps/server/src/ws.ts +++ b/apps/server/src/ws.ts @@ -67,6 +67,7 @@ import { WsCoreRpcGroup, WsOutsideAgentsRpcGroup, WsRpcGroup, + WsMovesRpcGroup, WsThemeShowcaseRpcGroup, WsThemesRpcGroup, WsOwnerRequiredError, @@ -176,6 +177,7 @@ import { ComputerLinks } from "./links/ComputerLinks.ts"; import { ScheduledTasks, snapshotForClient } from "./scheduled/ScheduledTasks.ts"; import { ThemeLibrary } from "./themes/ThemeLibrary.ts"; import { ThemeShowcase } from "./themes/ThemeShowcase.ts"; +import { MoveQueue } from "./moves/MoveQueue.ts"; import { ThreadMoves } from "./moves/ThreadMoves.ts"; import { ProjectSetups } from "./projectSetup/ProjectSetups.ts"; import { ProjectSetupSource } from "./projectSetup/ProjectSetupSource.ts"; @@ -735,6 +737,34 @@ const ThemeShowcaseRpcLayer = WsThemeShowcaseRpcGroup.toLayer( }), ); +/** + * Moving a thread to a linked computer (docs/design/fleet.md, Parts 3 and + * 3b): its own RPC group and layer (WsMovesRpcGroup), merged in + * makeWsRpcLayer. Open to every paired device, as the thread's other + * commands are. + */ +const MovesRpcLayer = WsMovesRpcGroup.toLayer( + Effect.gen(function* () { + const threadMoves = yield* ThreadMoves; + const moveQueue = yield* MoveQueue; + const aggregate = { "rpc.aggregate": "moves" } as const; + return WsMovesRpcGroup.of({ + [WS_METHODS.movesPreview]: (input) => + observeRpcEffect(WS_METHODS.movesPreview, threadMoves.preview(input.threadId), aggregate), + [WS_METHODS.movesStart]: (input) => + observeRpcEffect(WS_METHODS.movesStart, threadMoves.start(input), aggregate), + [WS_METHODS.movesCancel]: (input) => + observeRpcEffect(WS_METHODS.movesCancel, threadMoves.cancel(input.threadId), aggregate), + [WS_METHODS.movesQueue]: (input) => + observeRpcEffect(WS_METHODS.movesQueue, moveQueue.queue(input), aggregate), + [WS_METHODS.movesUnqueue]: (input) => + observeRpcEffect(WS_METHODS.movesUnqueue, moveQueue.unqueue(input), aggregate), + [WS_METHODS.movesRetry]: (input) => + observeRpcEffect(WS_METHODS.movesRetry, moveQueue.retry(input), aggregate), + }); + }), +); + /** Every RPC the WebSocket serves, for one connection. */ const makeWsRpcLayer = ( currentSession: { @@ -752,6 +782,7 @@ const makeWsRpcLayer = ( makeOutsideAgentsRpcLayer(currentSession.role), ThemesRpcLayer, ThemeShowcaseRpcLayer, + MovesRpcLayer, ); const makeCoreRpcLayer = ( @@ -779,7 +810,6 @@ const makeCoreRpcLayer = ( const pushAlerts = yield* PushAlerts; const computerLinks = yield* ComputerLinks; const scheduledTasks = yield* ScheduledTasks; - const threadMoves = yield* ThreadMoves; const projectSetups = yield* ProjectSetups; const projectSetupSource = yield* ProjectSetupSource; const projectionSnapshotQuery = yield* ProjectionSnapshotQuery; @@ -3484,19 +3514,7 @@ const makeCoreRpcLayer = ( scheduledTasks.runWithWebhookEvent(input), { "rpc.aggregate": "scheduledTasks" }, ), - // Moving a thread to a linked computer (docs/design/fleet.md, Part 3). - [WS_METHODS.movesPreview]: (input) => - observeRpcEffect(WS_METHODS.movesPreview, threadMoves.preview(input.threadId), { - "rpc.aggregate": "moves", - }), - [WS_METHODS.movesStart]: (input) => - observeRpcEffect(WS_METHODS.movesStart, threadMoves.start(input), { - "rpc.aggregate": "moves", - }), - [WS_METHODS.movesCancel]: (input) => - observeRpcEffect(WS_METHODS.movesCancel, threadMoves.cancel(input.threadId), { - "rpc.aggregate": "moves", - }), + // Setting up a project on another computer (docs/design/fleet.md, // Part 8): the app reads from the computer that has it and starts // the job on the one that gets the copy. From dfad3a3013c68d415d66f77411fd775e7fee9024 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:01:46 -0500 Subject: [PATCH 07/48] feat(server): an agent can move its own thread with thread_move A room tool on the main runtime, in the changes tier: it queues the caller's thread to move to a linked computer once its turn ends, with the target's suggested agent unless it names one. Refused for side runtimes, room agents, threads a linked computer or an outside agent started, and outside the caller's turn; a retried call finds the move it queued. --- apps/server/src/mcp/McpRoomServer.ts | 12 +++ apps/server/src/mcp/roomToolAccess.ts | 6 +- apps/server/src/mcp/threadToolHandlers.ts | 118 ++++++++++++++++++++++ apps/server/src/mcp/threadTools.ts | 29 ++++++ 4 files changed, 164 insertions(+), 1 deletion(-) diff --git a/apps/server/src/mcp/McpRoomServer.ts b/apps/server/src/mcp/McpRoomServer.ts index d0e1e5680..7a0de9b5a 100644 --- a/apps/server/src/mcp/McpRoomServer.ts +++ b/apps/server/src/mcp/McpRoomServer.ts @@ -45,6 +45,8 @@ import { ComputerLinkRepository } from "../links/ComputerLinkRepository.ts"; import { makeThreadToolLinks } from "../links/threadToolLinks.ts"; import { RepositoryIdentityResolver } from "../project/Services/RepositoryIdentityResolver.ts"; import { ScheduledTasks } from "../scheduled/ScheduledTasks.ts"; +import { MoveQueue } from "../moves/MoveQueue.ts"; +import { ThreadMoves } from "../moves/ThreadMoves.ts"; import { listOnlyGrantedTools, MCP_ROOM_ROUTE_PATH, @@ -106,6 +108,8 @@ const ROOM_TOOL_REACH = { thread_read: "project", thread_send: "changes", thread_stop: "changes", + // Moves only the caller's own thread, but off this computer. + thread_move: "changes", schedule_list: "project", schedule_create: "changes", schedule_update: "changes", @@ -241,6 +245,9 @@ const RoomToolHandlersLive = RoomEndpointToolkit.toLayer( }); // Scheduled tasks, when this server has them (docs/design/fleet.md, Part 6). const scheduledTasks = yield* Effect.serviceOption(ScheduledTasks); + // Moving a thread to a linked computer, when this server has links (Part 3b). + const threadMoves = yield* Effect.serviceOption(ThreadMoves); + const moveQueue = yield* Effect.serviceOption(MoveQueue); // The user's default for new threads: the last reasoning an agent-made thread follows. const userDefaultModel = settings.getSettings.pipe( Effect.map((current) => current.newThreadModelSelection), @@ -278,6 +285,9 @@ const RoomToolHandlersLive = RoomEndpointToolkit.toLayer( ...(Option.isSome(scheduledTasks) ? { scheduleAnswerWaiting: scheduledTasks.value.waitingThreadIds } : {}), + ...(Option.isSome(threadMoves) && Option.isSome(moveQueue) + ? { moves: { preview: threadMoves.value.preview, queue: moveQueue.value.queue } } + : {}), }); const scheduleHandlers = makeScheduleToolHandlers({ scheduledTasks: Option.getOrUndefined(scheduledTasks), @@ -483,6 +493,8 @@ const RoomToolHandlersLive = RoomEndpointToolkit.toLayer( changes("thread_send", (scope) => threadHandlers.thread_send(scope, input)), thread_stop: (input) => changes("thread_stop", (scope) => threadHandlers.thread_stop(scope, input)), + thread_move: (input) => + changes("thread_move", (scope) => threadHandlers.thread_move(scope, input)), schedule_list: (input) => projectRead( "schedule_list", diff --git a/apps/server/src/mcp/roomToolAccess.ts b/apps/server/src/mcp/roomToolAccess.ts index 6b0788f8a..cb0c7b669 100644 --- a/apps/server/src/mcp/roomToolAccess.ts +++ b/apps/server/src/mcp/roomToolAccess.ts @@ -18,7 +18,10 @@ /** The room endpoint's name, as each provider namespaces its tools. */ export const ROOM_MCP_SERVER_NAME = "threadlines_room"; -/** Tools for starting, reading, messaging and stopping other threads. */ +/** + * Tools for starting, reading, messaging and stopping other threads, and + * moving the caller's own thread to a linked computer (`thread_move`). + */ export const THREAD_TOOL_NAMES = [ "thread_agents", "thread_start", @@ -26,6 +29,7 @@ export const THREAD_TOOL_NAMES = [ "thread_read", "thread_send", "thread_stop", + "thread_move", ] as const; /** Tools for listing, creating and editing this computer's scheduled tasks. */ diff --git a/apps/server/src/mcp/threadToolHandlers.ts b/apps/server/src/mcp/threadToolHandlers.ts index b95a79bfb..17269198b 100644 --- a/apps/server/src/mcp/threadToolHandlers.ts +++ b/apps/server/src/mcp/threadToolHandlers.ts @@ -44,6 +44,10 @@ import { type RuntimeMode, type ServerProvider, ThreadId, + type ThreadMoveError, + type ThreadMovePreview, + type ThreadMoveQueueInput, + type ThreadMoveRequester, TurnId, } from "@threadlines/contracts"; import { placeMarkdownLink } from "@threadlines/shared/appPlaces"; @@ -91,6 +95,7 @@ import { THREAD_TITLE_CHAR_LIMIT, } from "../threads/threadOperations.ts"; import type { GitVcsDriverShape } from "../vcs/GitVcsDriver.ts"; +import { latchCallerTurn } from "./callerTurn.ts"; import type { McpInvocationScope } from "./McpSessionRegistry.ts"; import type { RoomRequestRegistry } from "./roomRequests.ts"; import type { RoomToolName } from "./roomToolAccess.ts"; @@ -99,6 +104,7 @@ import { roomCheckoutCwd } from "./roomToolHandlers.ts"; import type { ThreadAgentsResult, ThreadListResult, + ThreadMoveResult, ThreadReadResult, ThreadSendResult, ThreadStartResult, @@ -142,6 +148,16 @@ export interface ThreadToolDeps { * (docs/design/fleet.md, Part 6); absent on a server without scheduled tasks. */ readonly scheduleAnswerWaiting?: Effect.Effect>; + /** Moving a thread to a linked computer (docs/design/fleet.md, Part 3b); absent without links. */ + readonly moves?: ThreadToolMoves; +} + +/** What `thread_move` needs of the moves (ThreadMoves.preview, MoveQueue.queue). */ +export interface ThreadToolMoves { + readonly preview: (threadId: ThreadId) => Effect.Effect; + readonly queue: ( + input: ThreadMoveQueueInput & { readonly requestedBy: ThreadMoveRequester }, + ) => Effect.Effect; } /** A computer this one links to, as the thread tools see it. */ @@ -2013,6 +2029,107 @@ export function makeThreadToolHandlers(deps: ThreadToolDeps) { ); }); + // --------------------------------------------------------------------------- + // thread_move (docs/design/fleet.md, Part 3b) + // --------------------------------------------------------------------------- + + /** + * Queues the caller's own thread to move once its turn ends: the turn is + * the live work, so the move waits for the agent's last words. Only the + * thread's own agent, never a thread an outside agent or a linked computer + * started; the target's own Ask first, Allowed or Off still applies. + */ + const threadMove = ( + scope: McpInvocationScope, + input: { + readonly computer: string; + readonly agent?: { readonly instanceId: string; readonly model: string } | undefined; + }, + ): Effect.Effect => + Effect.gen(function* () { + const gated = yield* gate(scope, "thread_move"); + if ("refusal" in gated) return gated.refusal satisfies ThreadMoveResult; + const { thread } = gated; + if (scope.participantId !== null) { + return { + outcome: "not_allowed", + detail: "Only the thread's own agent can move it.", + } satisfies ThreadMoveResult; + } + if ((thread.outsideOrigin ?? null) !== null) { + return { + outcome: "not_allowed", + detail: + "An outside agent started this thread, so it can't move itself. The user can move it from the thread's menu.", + } satisfies ThreadMoveResult; + } + if (deps.moves === undefined) { + return { + outcome: "refused", + detail: "This computer can't move threads to other computers.", + } satisfies ThreadMoveResult; + } + const resolved = yield* resolveComputer(input.computer.trim()); + if ("refused" in resolved) { + return { outcome: "refused", detail: resolved.refused } satisfies ThreadMoveResult; + } + const { computer } = resolved; + const latched = yield* latchCallerTurn(deps.readThread, scope); + if (!("turnId" in latched)) { + return { + outcome: "refused", + detail: "Call thread_move during your turn.", + } satisfies ThreadMoveResult; + } + const queuedDetail = `Queued: this thread moves to ${computer.label} when your turn ends. Finish your reply; don't start more work.`; + // A retried call finds the move it queued. + const pending = latched.thread.pendingMove ?? null; + if ( + pending !== null && + pending.target.environmentId === computer.environmentId && + pending.requestedBy.kind === "agent" && + pending.requestedBy.turnId === latched.turnId + ) { + return { outcome: "queued", detail: queuedDetail } satisfies ThreadMoveResult; + } + const preview = yield* deps.moves.preview(thread.id).pipe(Effect.result); + if (preview._tag === "Failure") { + return { outcome: "refused", detail: preview.failure.detail } satisfies ThreadMoveResult; + } + if (preview.success.refusal !== undefined) { + return { outcome: "refused", detail: preview.success.refusal } satisfies ThreadMoveResult; + } + const target = preview.success.targets.find( + (entry) => entry.environmentId === computer.environmentId, + ); + const agent: ModelSelection | undefined = + input.agent !== undefined + ? ({ + instanceId: ProviderInstanceId.make(input.agent.instanceId.trim()), + model: input.agent.model.trim(), + } as ModelSelection) + : target?.suggestedAgent; + if (agent === undefined) { + return { + outcome: "refused", + detail: `${computer.label} can't be reached right now, so there's no agent to suggest there. Name one of its agents from thread_agents' \`computers\`, or try again once it connects.`, + } satisfies ThreadMoveResult; + } + const queued = yield* deps.moves + .queue({ + threadId: thread.id, + targetEnvironmentId: computer.environmentId, + agent, + mode: "when-done", + requestedBy: { kind: "agent", turnId: latched.turnId }, + }) + .pipe(Effect.result); + if (queued._tag === "Failure") { + return { outcome: "refused", detail: queued.failure.detail } satisfies ThreadMoveResult; + } + return { outcome: "queued", detail: queuedDetail } satisfies ThreadMoveResult; + }); + return { thread_agents: threadAgents, thread_start: threadStart, @@ -2020,6 +2137,7 @@ export function makeThreadToolHandlers(deps: ThreadToolDeps) { thread_read: threadRead, thread_send: threadSend, thread_stop: threadStop, + thread_move: threadMove, }; } diff --git a/apps/server/src/mcp/threadTools.ts b/apps/server/src/mcp/threadTools.ts index 0ae0a6f17..4afbf318d 100644 --- a/apps/server/src/mcp/threadTools.ts +++ b/apps/server/src/mcp/threadTools.ts @@ -152,6 +152,13 @@ export const ThreadStopResult = Schema.Struct({ }); export type ThreadStopResult = typeof ThreadStopResult.Type; +export const ThreadMoveResult = Schema.Struct({ + /** `queued`: it moves once this turn ends. */ + outcome: Schema.Literals(["queued", "off", "not_allowed", "refused"]), + detail: Schema.optional(Schema.String), +}); +export type ThreadMoveResult = typeof ThreadMoveResult.Type; + const ThreadIdParameter = Schema.String.annotate({ description: "The thread's id, from thread_start or thread_list.", }); @@ -322,6 +329,27 @@ export const ThreadStopTool = drivesThreads( }).annotate(Tool.Title, "Stop a thread"), ); +export const ThreadMoveTool = drivesThreads( + Tool.make("thread_move", { + description: + "Move this thread to another linked computer, to carry on there (the user may be shutting this one down). It moves when your turn ends, with its messages and code changes; you continue there in a fresh session with the conversation so far. Only your own thread. `computer` is a name or id from thread_agents' `computers`; `agent` is one of that computer's agents, left out for the one it suggests. The user can cancel it, and that computer may ask its user first. After calling it, finish your reply and don't start more work.", + parameters: Schema.Struct({ + computer: Schema.String.annotate({ + description: "The linked computer: a name or id from thread_agents' `computers`.", + }), + agent: Schema.optional( + Schema.Struct({ + instanceId: Schema.String.annotate({ + description: "From thread_agents' `computers`.", + }), + model: Schema.String.annotate({ description: "From thread_agents' `computers`." }), + }), + ), + }), + success: ThreadMoveResult, + }).annotate(Tool.Title, "Move this thread to another computer"), +); + export const ThreadToolkit = Toolkit.make( ThreadAgentsTool, ThreadStartTool, @@ -329,4 +357,5 @@ export const ThreadToolkit = Toolkit.make( ThreadReadTool, ThreadSendTool, ThreadStopTool, + ThreadMoveTool, ); From d40501e20d09df19693b4aeaed231e428260df63 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:17:51 -0500 Subject: [PATCH 08/48] docs(guide): agents can explain moving a thread to another computer The Fleet topic now says how Move to another computer works, what goes with the thread, what the other computer's user sees, and which threads can't move yet. --- apps/server/src/guide/topics.ts | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/apps/server/src/guide/topics.ts b/apps/server/src/guide/topics.ts index 3d5872e47..e9267e276 100644 --- a/apps/server/src/guide/topics.ts +++ b/apps/server/src/guide/topics.ts @@ -102,12 +102,26 @@ A thread on another computer stays under its parent on both computers and shows { id: "linking-computers", title: "Linking computers (Fleet)", - keywords: ["fleet", "link", "computer", "pc", "mac", "windows", "remote", "another machine"], + keywords: [ + "fleet", + "link", + "computer", + "pc", + "mac", + "windows", + "remote", + "another machine", + "move", + "transfer", + "shut down", + ], body: `Fleet, in the left bar, lists the computers this app knows. Use + at the top of its Computers list to link another computer running Threadlines. Each computer's page shows its connection, {place:fleet.worksWith}, and its devices. Linking is two-way, with a permission for each direction: "'s agents can start work on " is Allowed, Ask first or Off, with the most access those threads get ("Up to Auto"). Each direction is stored on the computer the work would run on, so the Mac's setting decides what the PC's agents may do on the Mac, and the other way round. Unlink from the same group. -With a link allowed, an agent can start a thread on the other computer in that computer's copy of the project, read it and message it, and its reply comes back. A computer you only open from this app (Settings › Connections › Computers you use from here) is not a link: you see its threads here, but agents can't work across. This app's sign-in to it renews whenever they connect and ends after 90 days without one, or when that computer removes it. Fleet's All computers page and its own page then say the sign-in ended and offer to connect again with a new code; its threads stay on it.`, +With a link allowed, an agent can start a thread on the other computer in that computer's copy of the project, read it and message it, and its reply comes back. A computer you only open from this app (Settings › Connections › Computers you use from here) is not a link: you see its threads here, but agents can't work across. This app's sign-in to it renews whenever they connect and ends after 90 days without one, or when that computer removes it. Fleet's All computers page and its own page then say the sign-in ended and offer to connect again with a new code; its threads stay on it. + +To carry on a thread on a linked computer, for example before shutting this one down, right-click it in the sidebar (or use the command palette) and pick Move to another computer. The thread goes with its messages and its code changes, including files not yet committed; staged changes arrive unstaged. It lands in a new worktree of that computer's copy of the project; the dialog offers to set the project up there when it's missing. When that computer's permission for this one is Ask first, its user allows the move there. The copy left here stays, read-only, with a link to the new one. On the other computer the agent starts a fresh session with the conversation so far, and turns from before the move can't be undone there. A thread that is working, waiting on you, in a room, that started other threads, or that has a goal, a check-back or pull request auto-fix on can't move yet; the dialog says what to finish first. General Chats don't move.`, }, { id: "devices", From 07ec3fc854249077b9482afa3ff777a0475aff85 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:02:02 -0500 Subject: [PATCH 09/48] docs(guide): agents can explain moving a working thread The Fleet topic's move paragraph now covers Move when it's done, Stop and move now, the notice while a move waits, and an agent moving its own thread with thread_move. --- apps/server/src/guide/topics.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/server/src/guide/topics.ts b/apps/server/src/guide/topics.ts index e9267e276..39a3f9f6b 100644 --- a/apps/server/src/guide/topics.ts +++ b/apps/server/src/guide/topics.ts @@ -121,7 +121,7 @@ Linking is two-way, with a permission for each direction: "'s agents can star With a link allowed, an agent can start a thread on the other computer in that computer's copy of the project, read it and message it, and its reply comes back. A computer you only open from this app (Settings › Connections › Computers you use from here) is not a link: you see its threads here, but agents can't work across. This app's sign-in to it renews whenever they connect and ends after 90 days without one, or when that computer removes it. Fleet's All computers page and its own page then say the sign-in ended and offer to connect again with a new code; its threads stay on it. -To carry on a thread on a linked computer, for example before shutting this one down, right-click it in the sidebar (or use the command palette) and pick Move to another computer. The thread goes with its messages and its code changes, including files not yet committed; staged changes arrive unstaged. It lands in a new worktree of that computer's copy of the project; the dialog offers to set the project up there when it's missing. When that computer's permission for this one is Ask first, its user allows the move there. The copy left here stays, read-only, with a link to the new one. On the other computer the agent starts a fresh session with the conversation so far, and turns from before the move can't be undone there. A thread that is working, waiting on you, in a room, that started other threads, or that has a goal, a check-back or pull request auto-fix on can't move yet; the dialog says what to finish first. General Chats don't move.`, +To carry on a thread on a linked computer, for example before shutting this one down, right-click it in the sidebar (or use the command palette) and pick Move to another computer. The thread goes with its messages and its code changes, including files not yet committed; staged changes arrive unstaged. It lands in a new worktree of that computer's copy of the project; the dialog offers to set the project up there when it's missing. When that computer's permission for this one is Ask first, its user allows the move there. The copy left here stays, read-only, with a link to the new one. On the other computer the agent starts a fresh session with the conversation so far, and turns from before the move can't be undone there. A working thread can move too: Move when it's done starts the move once its turn and background work finish, and Stop and move now stops its agent first. While it waits, a notice above the message box says what for, with Cancel, and the thread takes no new messages. An agent can move its own thread with thread_move; it goes when that turn ends. A thread in a room, that started other threads, or that has a goal, a check-back, queued messages or pull request auto-fix on can't move yet; the dialog says what to finish first. General Chats don't move.`, }, { id: "devices", From e626f5b0961cb9da722ab19cb6e0719c4ea91094 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:07:15 -0500 Subject: [PATCH 10/48] test(server): queued moves, the session fence and thread_move MoveQueue runs against the real decider and projector: a move starts once the turn completes, waits out background tasks, ends when the agent set a check-back, holds while the target is offline and retries on its own and on reconnect, holds while a stop can't be proven, survives a restart, goes back to waiting when a provider turn slipped in, and a Cancel that lost the race cancels the transfer. ProviderService refuses to start or recover any session of a fenced thread and needs each driver's proof to quiesce one. thread_move queues with the target's suggested agent and is refused for side runtimes, room agents and threads another computer or an outside agent started. --- .../server/src/mcp/threadToolHandlers.test.ts | 119 +++++- apps/server/src/moves/MoveQueue.test.ts | 391 ++++++++++++++++++ apps/server/src/moves/MoveQueue.ts | 23 +- .../provider/Layers/ProviderService.test.ts | 134 +++++- 4 files changed, 662 insertions(+), 5 deletions(-) create mode 100644 apps/server/src/moves/MoveQueue.test.ts diff --git a/apps/server/src/mcp/threadToolHandlers.test.ts b/apps/server/src/mcp/threadToolHandlers.test.ts index bf0b626af..f7ff2f2de 100644 --- a/apps/server/src/mcp/threadToolHandlers.test.ts +++ b/apps/server/src/mcp/threadToolHandlers.test.ts @@ -7,6 +7,7 @@ import { type PeerDescriptor, EMPTY_CHILD_REQUEST_STATE, MessageId, + OutsideAgentConnectionId, type OrchestrationChildRequest, type OrchestrationCommand, type OrchestrationMessage, @@ -27,7 +28,11 @@ import type { McpInvocationScope } from "./McpSessionRegistry.ts"; import { makeRoomRequestRegistry } from "./roomRequests.ts"; import { roomToolsFor, THREAD_TOOL_NAMES } from "./roomToolAccess.ts"; import { linkThreadId } from "../links/linkIds.ts"; -import { makeThreadToolHandlers, type ThreadToolLinks } from "./threadToolHandlers.ts"; +import { + makeThreadToolHandlers, + type ThreadToolLinks, + type ThreadToolMoves, +} from "./threadToolHandlers.ts"; const PROJECT = ProjectId.make("project-1"); const PARENT = ThreadId.make("0f8f5a52-4d0e-4c1b-9d56-1e7c1f9b7a01"); @@ -204,6 +209,7 @@ const makeTools = (options: { readonly shareWorktrees?: boolean; /** The branch checked out in a folder; any other folder has none. */ readonly branches?: Readonly>; + readonly moves?: ThreadToolMoves; }) => Effect.gen(function* () { let caller = options.caller ?? makeThread(); @@ -282,6 +288,7 @@ const makeTools = (options: { }, requests: makeRoomRequestRegistry(yield* Effect.scope), ...(options.links !== undefined ? { links: options.links } : {}), + ...(options.moves !== undefined ? { moves: options.moves } : {}), }); return { handlers, @@ -1377,3 +1384,113 @@ describe("thread tools across linked computers", () => { }), ); }); + +describe("thread_move", () => { + /** Moves that record what was queued, to a Mac mini that suggests its Claude. */ + const fakeMoves = () => { + const queued: Array[0]> = []; + const moves: ThreadToolMoves = { + preview: () => + Effect.succeed({ + targets: [ + { + environmentId: MAC_MINI, + label: "Mac mini", + agents: MAC_DESCRIPTOR.agents, + suggestedAgent: { + instanceId: ProviderInstanceId.make("mac-claude"), + model: "opus-5-5", + }, + }, + ], + }), + queue: (input) => + Effect.sync(() => { + queued.push(input); + return { requestId: "request-1" }; + }), + }; + return { moves, queued }; + }; + + it.effect("queues the caller's own thread to move when its turn ends", () => + Effect.gen(function* () { + const { moves, queued } = fakeMoves(); + const tools = yield* makeTools({ + links: fakeLinks(), + moves, + caller: makeThread({ deletedAt: null }), + }); + const result = yield* tools.handlers.thread_move(mainCaller(), { computer: "mac mini" }); + expect(result).toEqual({ + outcome: "queued", + detail: + "Queued: this thread moves to Mac mini when your turn ends. Finish your reply; don't start more work.", + }); + expect(queued).toEqual([ + { + threadId: PARENT, + targetEnvironmentId: MAC_MINI, + agent: { instanceId: "mac-claude", model: "opus-5-5" }, + mode: "when-done", + requestedBy: { kind: "agent", turnId: TURN }, + }, + ]); + }), + ); + + it.effect( + "is refused for side runtimes, room agents, and threads another computer or an outside agent started", + () => + Effect.gen(function* () { + const tryAs = (scope: McpInvocationScope, caller?: OrchestrationThread) => + Effect.gen(function* () { + const { moves, queued } = fakeMoves(); + const tools = yield* makeTools({ + links: fakeLinks(), + moves, + ...(caller ? { caller } : {}), + }); + const result = yield* tools.handlers.thread_move(scope, { computer: "Mac mini" }); + expect(queued).toHaveLength(0); + return result.outcome; + }); + const side: McpInvocationScope = { + ...mainCaller(), + side: { + sideTurnId: SideTurnId.make("11111111-2222-4333-8444-555555555555"), + kind: "ask", + }, + }; + const roomAgent: McpInvocationScope = { + ...mainCaller(), + participantId: ThreadParticipantId.make("0b5c2a6e-9f1d-4e8a-8b7c-6d5e4f3a2b1c"), + }; + expect([ + yield* tryAs(side), + yield* tryAs(roomAgent), + yield* tryAs( + mainCaller(), + makeThread({ + remoteOrigin: { + linkDeviceId: "device-1", + environmentId: MAC_MINI, + environmentLabel: "Mac mini", + threadId: ThreadId.make("thread-there"), + threadTitle: "Fix it there", + }, + }), + ), + yield* tryAs( + mainCaller(), + makeThread({ + outsideOrigin: { + connectionId: OutsideAgentConnectionId.make("connection-1"), + name: "Grok Bot", + }, + }), + ), + ]).toEqual(["not_allowed", "not_allowed", "not_allowed", "not_allowed"]); + }), + ); +}); diff --git a/apps/server/src/moves/MoveQueue.test.ts b/apps/server/src/moves/MoveQueue.test.ts new file mode 100644 index 000000000..8d78fec4e --- /dev/null +++ b/apps/server/src/moves/MoveQueue.test.ts @@ -0,0 +1,391 @@ +import { + CommandId, + EnvironmentId, + ProjectId, + ProviderInstanceId, + ThreadId, + ThreadMoveError, + ThreadMoveId, + TurnId, + type ModelSelection, + type OrchestrationCommand, + type OrchestrationEvent, + type OrchestrationReadModel, + type OrchestrationSession, + type ThreadMoveRequestId, +} from "@threadlines/contracts"; +import * as Duration from "effect/Duration"; +import * as Effect from "effect/Effect"; +import * as PubSub from "effect/PubSub"; +import * as Scope from "effect/Scope"; +import * as Stream from "effect/Stream"; +import { describe, expect, it } from "vite-plus/test"; + +import { decideOrchestrationCommand } from "../orchestration/decider.ts"; +import { createEmptyReadModel, projectEvent } from "../orchestration/projector.ts"; +import type { ProviderStopConfirmation } from "../provider/Services/ProviderAdapter.ts"; +import { makeMoveQueue, type MoveQueueShape } from "./MoveQueue.ts"; +import type { QueuedMoveStart } from "./ThreadMoves.ts"; + +const now = "2026-11-15T10:00:00.000Z"; +const PROJECT = ProjectId.make("project-1"); +const THREAD = ThreadId.make("thread-1"); +const MAC_MINI = EnvironmentId.make("environment-mac-mini"); +const agent: ModelSelection = { + instanceId: ProviderInstanceId.make("codex"), + model: "gpt-6.1-sol", +}; + +let commandCount = 0; +const nextCommandId = () => CommandId.make(`cmd-${(commandCount += 1)}`); + +/** + * The real decider and projector behind a fake engine, with a fake provider + * side (a session stop lands as a stopped session, as ProviderCommandReactor + * does) and fake links. + */ +async function harness(options?: { readonly model?: OrchestrationReadModel }) { + const scope = await Effect.runPromise(Scope.make()); + const events = await Effect.runPromise(PubSub.unbounded()); + const linkChanges = await Effect.runPromise(PubSub.unbounded()); + let model = options?.model ?? (await seeded()); + let sequence = 0; + const state = { + online: true, + stopsLand: true, + confirmation: { confirmed: true } as ProviderStopConfirmation, + quiesceCalls: 0, + starts: 0, + cancels: 0, + /** Runs right before a start, as if a provider turn slipped in. */ + beforeStart: null as null | (() => Promise), + }; + + const dispatch = (command: OrchestrationCommand): Effect.Effect => + Effect.gen(function* () { + const decided = yield* decideOrchestrationCommand({ command, readModel: model }).pipe( + Effect.mapError((error) => new ThreadMoveError({ detail: error.message })), + ); + for (const planned of Array.isArray(decided) ? decided : [decided]) { + sequence += 1; + const event = { ...planned, sequence } as OrchestrationEvent; + model = yield* projectEvent(model, event).pipe(Effect.orDie); + yield* PubSub.publish(events, event); + if (event.type === "thread.session-stop-requested" && state.stopsLand) { + yield* setSession("stopped"); + } + } + }); + const setSession = ( + status: OrchestrationSession["status"], + extra?: Partial, + ) => + dispatch({ + type: "thread.session.set", + commandId: nextCommandId(), + threadId: THREAD, + session: { + threadId: THREAD, + status, + providerName: "codex", + runtimeMode: "auto", + activeTurnId: null, + lastError: null, + updatedAt: now, + ...extra, + }, + createdAt: now, + } as OrchestrationCommand); + + const thread = () => model.threads.find((entry) => entry.id === THREAD)!; + + const startQueued = (input: { + readonly threadId: ThreadId; + readonly requestId: ThreadMoveRequestId; + }): Effect.Effect => + Effect.gen(function* () { + if (state.beforeStart !== null) { + const run = state.beforeStart; + state.beforeStart = null; + yield* Effect.promise(run); + } + if (!state.online) { + return { kind: "unreachable", words: "Waiting for Mac mini to connect." } as const; + } + const transferId = ThreadMoveId.make(`transfer-${(state.starts += 1)}`); + const started = yield* dispatch({ + type: "thread.move.start", + commandId: nextCommandId(), + threadId: input.threadId, + transferId, + target: { environmentId: MAC_MINI, label: "Mac mini" }, + agent, + queuedRequestId: input.requestId, + createdAt: now, + }).pipe(Effect.result); + return started._tag === "Success" + ? ({ kind: "started", transferId } as const) + : ({ kind: "busy", words: started.failure.detail } as const); + }); + + const queue: MoveQueueShape = await Effect.runPromise( + makeMoveQueue({ + dispatch, + subscribeEvents: Effect.map(PubSub.subscribe(events), Stream.fromSubscription), + linkChanges: Stream.fromPubSub(linkChanges), + readThread: (threadId) => + Effect.sync(() => model.threads.find((entry) => entry.id === threadId) ?? null), + readModel: Effect.sync(() => model), + quiesce: () => + Effect.sync(() => { + state.quiesceCalls += 1; + return state.confirmation; + }), + checkQueue: () => Effect.succeed({ label: "Mac mini" }), + startQueued, + cancelMove: (threadId) => { + state.cancels += 1; + const move = model.threads.find((entry) => entry.id === threadId)?.move ?? null; + return move === null + ? Effect.void + : dispatch({ + type: "thread.move.cancel", + commandId: nextCommandId(), + threadId, + transferId: move.transferId, + createdAt: now, + }); + }, + retryDelay: () => Duration.millis(40), + settleDelay: Duration.millis(5), + }).pipe(Scope.provide(scope)), + ); + const run = (effect: Effect.Effect) => Effect.runPromise(effect); + return { + state, + queue, + thread, + model: () => model, + run, + setSession: (status: OrchestrationSession["status"], extra?: Partial) => + run(setSession(status, extra)), + dispatch: (command: OrchestrationCommand) => run(dispatch(command)), + /** The turn ends as a provider ends it: completed, then the session idles. */ + endTurn: (turnId: TurnId) => + run( + dispatch({ + type: "thread.turn.complete", + commandId: nextCommandId(), + threadId: THREAD, + turnId, + state: "completed", + completedAt: now, + createdAt: now, + } as OrchestrationCommand).pipe(Effect.andThen(setSession("ready"))), + ), + startReactor: () => run(queue.startReactor().pipe(Scope.provide(scope))), + linkChanged: () => run(PubSub.publish(linkChanges, undefined)), + close: () => run(Scope.close(scope, { _tag: "Success", value: undefined } as never)), + queueMove: (mode: "when-done" | "stop-now" = "when-done") => + run(queue.queue({ threadId: THREAD, targetEnvironmentId: MAC_MINI, agent, mode })), + }; +} + +async function seeded(): Promise { + let model = createEmptyReadModel(now); + let sequence = 0; + for (const command of [ + { + type: "project.create", + commandId: nextCommandId(), + projectId: PROJECT, + title: "Threadlines", + workspaceRoot: "/repo", + createdAt: now, + }, + { + type: "thread.create", + commandId: nextCommandId(), + threadId: THREAD, + projectId: PROJECT, + title: "Fix the flaky test", + modelSelection: agent, + runtimeMode: "auto", + interactionMode: "default", + branch: "fix/flaky", + worktreePath: "/worktrees/fix-flaky", + createdAt: now, + }, + ] as OrchestrationCommand[]) { + const decided = await Effect.runPromise( + decideOrchestrationCommand({ command, readModel: model }), + ); + for (const planned of Array.isArray(decided) ? decided : [decided]) { + sequence += 1; + model = await Effect.runPromise( + projectEvent(model, { ...planned, sequence } as OrchestrationEvent), + ); + } + } + return model; +} + +async function waitFor(check: () => boolean, what: string) { + const deadline = Date.now() + 3_000; + while (!check()) { + if (Date.now() > deadline) throw new Error(`timed out waiting: ${what}`); + await new Promise((resolve) => setTimeout(resolve, 10)); + } +} + +const settle = () => new Promise((resolve) => setTimeout(resolve, 150)); +const TURN = TurnId.make("turn-1"); + +describe("MoveQueue", () => { + it("starts the move once the turn completes, not before", async () => { + const h = await harness(); + await h.startReactor(); + await h.setSession("running", { activeTurnId: TURN }); + await h.queueMove(); + await settle(); + expect(h.thread().pendingMove).toMatchObject({ stage: "waiting" }); + expect(h.thread().pendingMove?.waitsFor?.map((wait) => wait.code)).toEqual(["turn"]); + expect(h.state.quiesceCalls).toBe(0); + + await h.endTurn(TURN); + await waitFor(() => (h.thread().move ?? null) !== null, "the move to start"); + expect(h.thread().pendingMove ?? null).toBeNull(); + expect(h.thread().move?.queuedRequestId).toBeDefined(); + expect(h.state.quiesceCalls).toBeGreaterThan(0); + await h.close(); + }); + + it("waits for background tasks to settle", async () => { + const h = await harness(); + await h.startReactor(); + await h.setSession("ready", { pendingBackgroundTaskCount: 1, awaitedBackgroundTaskCount: 1 }); + await h.queueMove(); + await settle(); + expect(h.thread().pendingMove?.waitsFor?.map((wait) => wait.code)).toEqual(["background"]); + expect(h.thread().move ?? null).toBeNull(); + + await h.setSession("ready", { pendingBackgroundTaskCount: 0, awaitedBackgroundTaskCount: 0 }); + await waitFor(() => (h.thread().move ?? null) !== null, "the move to start"); + await h.close(); + }); + + it("ends as blocked when the agent set a check-back during the turn", async () => { + const h = await harness(); + await h.startReactor(); + await h.setSession("running", { activeTurnId: TURN }); + await h.queueMove(); + await h.dispatch({ + type: "thread.check-backs.set", + commandId: nextCommandId(), + threadId: THREAD, + participantId: null, + checkBacks: [ + { + id: "check-back-1", + participantId: null, + scheduledFor: "2026-11-15T12:00:00.000Z", + exact: true, + recurring: false, + durable: false, + }, + ], + createdAt: now, + }); + await waitFor(() => (h.thread().pendingMove ?? null) === null, "the wait to end"); + expect(h.thread().move ?? null).toBeNull(); + expect(h.thread().activities.at(-1)).toMatchObject({ + kind: "thread.move-not-started", + payload: { + reason: "blocked", + words: "This thread's agent set itself a time to come back. Cancel it first.", + }, + }); + await h.close(); + }); + + it("waits while the target is offline, retries on its own, and starts once it connects", async () => { + const h = await harness(); + h.state.online = false; + await h.startReactor(); + await h.queueMove(); + await waitFor( + () => h.thread().pendingMove?.hold?.kind === "target_unreachable", + "the offline hold", + ); + expect(h.thread().pendingMove?.hold?.words).toBe("Waiting for Mac mini to connect."); + const quiescedWhileOffline = h.state.quiesceCalls; + // A failed describe on a connected link is tried again without any change. + await waitFor(() => h.state.quiesceCalls > quiescedWhileOffline, "a retry on its own"); + expect(h.thread().move ?? null).toBeNull(); + + h.state.online = true; + await h.linkChanged(); + await waitFor(() => (h.thread().move ?? null) !== null, "the move to start on reconnect"); + await h.close(); + }); + + it("never moves while a stop can't be proven, and says why", async () => { + const h = await harness(); + h.state.confirmation = { confirmed: false, reason: "its process 4242 didn't exit" }; + await h.startReactor(); + await h.queueMove(); + await waitFor(() => h.thread().pendingMove?.hold?.kind === "stop_unconfirmed", "the hold"); + expect(h.thread().pendingMove?.hold?.words).toBe( + "Couldn't confirm the agent stopped: its process 4242 didn't exit.", + ); + expect(h.thread().pendingMove?.stage).toBe("quiescing"); + await settle(); + expect(h.thread().move ?? null).toBeNull(); + + h.state.confirmation = { confirmed: true }; + await h.run(h.queue.retry({ threadId: THREAD, requestId: h.thread().pendingMove!.requestId })); + await waitFor(() => (h.thread().move ?? null) !== null, "the move after Try again"); + await h.close(); + }); + + it("picks up a pending move the last process left", async () => { + const first = await harness(); + await first.setSession("running", { activeTurnId: TURN }); + await first.queueMove(); + const left = first.model(); + await first.close(); + + const restarted = await harness({ model: left }); + await restarted.startReactor(); + await restarted.endTurn(TURN); + await waitFor(() => (restarted.thread().move ?? null) !== null, "the move after a restart"); + await restarted.close(); + }); + + it("goes back to waiting when a provider turn slipped in after the thread looked free", async () => { + const h = await harness(); + // The turn the provider started on its own lands just before the start. + h.state.beforeStart = () => h.setSession("running", { activeTurnId: TurnId.make("turn-own") }); + await h.startReactor(); + await h.queueMove(); + await settle(); + expect(h.thread().move ?? null).toBeNull(); + expect(h.thread().pendingMove?.waitsFor?.map((wait) => wait.code)).toEqual(["turn"]); + + await h.endTurn(TurnId.make("turn-own")); + await waitFor(() => (h.thread().move ?? null) !== null, "the move after that turn"); + await h.close(); + }); + + it("a Cancel that lost the race to the start cancels the transfer instead", async () => { + const h = await harness(); + await h.startReactor(); + const { requestId } = await h.queueMove(); + await waitFor(() => (h.thread().move ?? null) !== null, "the move to start"); + // The user's Cancel names the queued move it showed. + await h.run(h.queue.unqueue({ threadId: THREAD, requestId })); + expect(h.state.cancels).toBe(1); + expect(h.thread().move?.cancelRequested).toBe(true); + await h.close(); + }); +}); diff --git a/apps/server/src/moves/MoveQueue.ts b/apps/server/src/moves/MoveQueue.ts index 2f94abb02..50b839ea9 100644 --- a/apps/server/src/moves/MoveQueue.ts +++ b/apps/server/src/moves/MoveQueue.ts @@ -106,6 +106,14 @@ const defaultRetryDelay = (attempt: number) => /** How long after a quiesce it is asked again for a session the projection still shows. */ const REQUIESCE_AFTER_MS = 10_000; +/** A session as projected, to tell whether it changed since the last quiesce. */ +const sessionMark = (thread: OrchestrationThread) => { + const session = thread.session; + return session === null + ? "none" + : `${session.status}:${session.activeTurnId ?? ""}:${session.updatedAt}`; +}; + /** A thread's move checks, from the thread alone: no other thread or project. */ const localRefusals = (thread: OrchestrationThread) => threadMoveRefusals({ projects: [], threads: [thread] }, thread); @@ -119,6 +127,8 @@ interface TrackedMove { /** How many times in a row it was held (unreachable, unconfirmed). */ holds: number; quiescedAtMs: number; + /** The session the last quiesce saw (sessionMark). */ + quiescedSession: string; } export const makeMoveQueue = ( @@ -242,13 +252,14 @@ export const makeMoveQueue = ( /** Watches a thread's queued move `requestId`, replacing an older one. */ const track = (threadId: ThreadId, requestId: ThreadMoveRequestId) => - Effect.gen(function* () { + Effect.sync(() => { const existing = tracked.get(threadId); if (existing !== undefined) { if (existing.requestId === requestId) return; existing.requestId = requestId; existing.holds = 0; existing.quiescedAtMs = 0; + existing.quiescedSession = ""; return; } tracked.set(threadId, { @@ -259,6 +270,7 @@ export const makeMoveQueue = ( timerDueMs: 0, holds: 0, quiescedAtMs: 0, + quiescedSession: "", }); }); @@ -320,8 +332,15 @@ export const makeMoveQueue = ( // The barrier: the fence goes up and its sessions stop. if (pending.stage === "waiting" || !sessionStopped(thread)) { - if (pending.stage === "waiting" || Date.now() - entry.quiescedAtMs > REQUIESCE_AFTER_MS) { + // Asked again once the session changed since (a turn that slipped + // in and ended), or the stop never showed. + if ( + pending.stage === "waiting" || + sessionMark(thread) !== entry.quiescedSession || + Date.now() - entry.quiescedAtMs > REQUIESCE_AFTER_MS + ) { entry.quiescedAtMs = Date.now(); + entry.quiescedSession = sessionMark(thread); yield* dispatchIgnoringRefusal({ type: "thread.move.quiesce", commandId: commandId(requestId, "quiesce"), diff --git a/apps/server/src/provider/Layers/ProviderService.test.ts b/apps/server/src/provider/Layers/ProviderService.test.ts index ff62b72ab..278cf9836 100644 --- a/apps/server/src/provider/Layers/ProviderService.test.ts +++ b/apps/server/src/provider/Layers/ProviderService.test.ts @@ -22,10 +22,11 @@ import { ProviderSessionStartInput, SideTurnId, ThreadId, + ThreadParticipantId, TurnId, } from "@threadlines/contracts"; import { createModelSelection } from "@threadlines/shared/model"; -import { sideSessionKey } from "@threadlines/shared/threadParticipants"; +import { participantSessionKey, sideSessionKey } from "@threadlines/shared/threadParticipants"; import { it, assert, vi } from "@effect/vitest"; import * as Clock from "effect/Clock"; @@ -54,6 +55,7 @@ import { import type { ProviderAdapterShape, ProviderSetThreadGoalInput, + ProviderStopConfirmation, } from "../Services/ProviderAdapter.ts"; import { ProviderAdapterRegistry, @@ -74,7 +76,10 @@ import { import { ServerSettingsService } from "../../serverSettings.ts"; import { AnalyticsService } from "../../telemetry/Services/AnalyticsService.ts"; import { makeAdapterRegistryMock } from "../testUtils/providerAdapterRegistryMock.ts"; -import { ThreadSessionAdmissionPermissive } from "../Services/ThreadSessionAdmission.ts"; +import { + ThreadSessionAdmission, + ThreadSessionAdmissionPermissive, +} from "../Services/ThreadSessionAdmission.ts"; const defaultServerSettingsLayer = ServerSettingsService.layerTest(); @@ -3051,3 +3056,128 @@ validation.layer("ProviderServiceLive validation", (it) => { }), ); }); + +/** + * The session fence and the proven stop (docs/design/fleet.md, Part 3b), + * against an admission whose fenced threads the test sets. + */ +function makeFencedProviderServiceLayer() { + const codex = makeFakeCodexAdapter(); + const claude = makeFakeCodexAdapter(CLAUDE_AGENT_DRIVER); + const fenced = new Set(); + const confirmThreadStopped = vi.fn( + (_threadId: ThreadId): Effect.Effect => + Effect.succeed({ confirmed: true }), + ); + const registry = makeAdapterRegistryMock({ + [ProviderDriverKind.make("codex")]: { ...codex.adapter, confirmThreadStopped }, + // A driver that can't prove its stop. + [ProviderDriverKind.make("claudeAgent")]: claude.adapter, + }); + const runtimeRepositoryLayer = ProviderSessionRuntimeRepositoryLive.pipe( + Layer.provide(SqlitePersistenceMemory), + ); + const directoryLayer = ProviderSessionDirectoryLive.pipe(Layer.provide(runtimeRepositoryLayer)); + const admission = Layer.succeed( + ThreadSessionAdmission, + ThreadSessionAdmission.of({ + refusal: (threadId) => + Effect.succeed(fenced.has(threadId) ? "This thread is moving to Mac mini." : null), + }), + ); + const layer = it.layer( + Layer.mergeAll( + makeProviderServiceLive().pipe( + Layer.provide(admission), + Layer.provide(Layer.succeed(ProviderAdapterRegistry, registry)), + Layer.provide(directoryLayer), + Layer.provide(defaultServerSettingsLayer), + Layer.provideMerge(AnalyticsService.layerTest), + Layer.provide(Layer.succeed(ProviderEventLoggers, NoOpProviderEventLoggers)), + ), + directoryLayer, + runtimeRepositoryLayer, + NodeServices.layer, + ), + ); + return { codex, claude, fenced, confirmThreadStopped, layer }; +} + +const fence = makeFencedProviderServiceLayer(); + +fence.layer("ProviderServiceLive moving threads", (it) => { + const MOVING = asThreadId("6f1d1e0a-7a51-4c3b-9d47-1a2b3c4d5e6f"); + const ROOM_AGENT = ThreadParticipantId.make("0b5c2a6e-9f1d-4e8a-8b7c-6d5e4f3a2b1c"); + const startOn = (instance: ProviderInstanceId, key: ThreadId) => + Effect.gen(function* () { + const provider = yield* ProviderService; + return yield* provider.startSession(key, { + provider: ProviderDriverKind.make(instance === codexInstanceId ? "codex" : "claudeAgent"), + providerInstanceId: instance, + threadId: key, + cwd: "/tmp/project", + runtimeMode: "full-access", + }); + }); + + it.effect("starts and recovers no session of a fenced thread, whatever its key", () => + Effect.gen(function* () { + const provider = yield* ProviderService; + const started = yield* startOn(codexInstanceId, MOVING); + // The runtime is gone; the binding would bring it back. + yield* fence.codex.stopSession(started.threadId); + fence.codex.startSession.mockClear(); + fence.fenced.add(MOVING); + + const recovered = yield* provider + .rollbackConversation({ threadId: MOVING, numTurns: 1 }) + .pipe(Effect.result); + assert.equal(recovered._tag, "Failure"); + const roomAgent = yield* startOn( + codexInstanceId, + participantSessionKey(MOVING, ROOM_AGENT), + ).pipe(Effect.result); + assert.equal(roomAgent._tag, "Failure"); + assert.include( + String(roomAgent._tag === "Failure" ? roomAgent.failure.message : ""), + "moving to Mac mini", + ); + assert.equal(fence.codex.startSession.mock.calls.length, 0); + + fence.fenced.delete(MOVING); + yield* provider.rollbackConversation({ threadId: MOVING, numTurns: 1 }); + assert.equal(fence.codex.startSession.mock.calls.length, 1); + }), + ); + + it.effect("quiesceThread stops every runtime of the thread and needs each driver's proof", () => + Effect.gen(function* () { + const provider = yield* ProviderService; + const other = asThreadId("thread-stays"); + yield* startOn(codexInstanceId, MOVING); + yield* startOn(codexInstanceId, participantSessionKey(MOVING, ROOM_AGENT)); + yield* startOn(codexInstanceId, other); + + assert.deepEqual(yield* provider.quiesceThread(MOVING), { confirmed: true }); + const left = (yield* provider.listSessions()).map((session) => session.threadId); + assert.deepEqual(left, [other]); + assert.deepEqual(fence.confirmThreadStopped.mock.calls.at(-1), [MOVING]); + + // A driver that can't prove its stop never passes for a thread it ran. + yield* startOn(ProviderInstanceId.make("claudeAgent"), MOVING); + const unproven = yield* provider.quiesceThread(MOVING); + assert.deepEqual(unproven, { + confirmed: false, + reason: "claudeAgent can't confirm it stopped", + }); + // Nor does one whose own proof failed. + fence.confirmThreadStopped.mockImplementationOnce(() => + Effect.succeed({ confirmed: false, reason: "its process 4242 didn't exit" }), + ); + assert.deepEqual(yield* provider.quiesceThread(MOVING), { + confirmed: false, + reason: "its process 4242 didn't exit", + }); + }), + ); +}); From d628d4adf5a31ef19f078260fd33624232e53aa5 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:08:31 -0500 Subject: [PATCH 11/48] docs(server): the move decisions say how a queued move ends --- apps/server/src/orchestration/threadMoveDecisions.ts | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/apps/server/src/orchestration/threadMoveDecisions.ts b/apps/server/src/orchestration/threadMoveDecisions.ts index 949015568..2f8da2ad2 100644 --- a/apps/server/src/orchestration/threadMoveDecisions.ts +++ b/apps/server/src/orchestration/threadMoveDecisions.ts @@ -5,8 +5,11 @@ * unfreezes (the other computer confirmed the move was dropped) or stays * read-only for good (`thread.moved-away`). On the computer it arrives at, * `thread.import` creates it with its whole history in one decision, so the - * thread existing is the move's commit. Each function returns the events to - * plan, or a refusal to report. + * thread existing is the move's commit. A move can also wait for its thread + * to be free (Part 3b): `thread.move.queue` records a pending move, the + * guard keeps new work off the thread meanwhile, and the move queue's + * `thread.move.start` ends the pending move as it freezes the thread. Each + * function returns the events to plan, or a refusal to report. */ import { EventId, From 10a23a6f47bb0990fca121e42444f3d57b8c2bf1 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:54:03 -0500 Subject: [PATCH 12/48] test(server): the agent-process ledger ends a shell an agent left in its own group --- .../src/provider/agentProcessLedger.test.ts | 82 +++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 apps/server/src/provider/agentProcessLedger.test.ts diff --git a/apps/server/src/provider/agentProcessLedger.test.ts b/apps/server/src/provider/agentProcessLedger.test.ts new file mode 100644 index 000000000..a419bf7b8 --- /dev/null +++ b/apps/server/src/provider/agentProcessLedger.test.ts @@ -0,0 +1,82 @@ +import { spawn } from "node:child_process"; +import { once } from "node:events"; + +import { ThreadId } from "@threadlines/contracts"; +import { it } from "@effect/vitest"; +import * as Effect from "effect/Effect"; +import { describe, expect } from "vite-plus/test"; + +import { makeAgentProcessLedger, realProcessProbe } from "./agentProcessLedger.ts"; + +const THREAD = ThreadId.make("thread-ledger"); + +// Starts a shell of its own (a process group it leads), prints its pid, and waits. +const LEADER = ` +const { spawn } = require("node:child_process"); +const shell = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { + detached: true, + stdio: "ignore", +}); +shell.unref(); +process.stdout.write(String(shell.pid) + "\\n"); +setInterval(() => {}, 1000); +`; + +const isRunning = (pid: number) => realProcessProbe.exists(pid); + +describe.skipIf(process.platform === "win32")("agent process ledger", () => { + it.live("ends what an agent left in a process group of its own, then confirms", () => + Effect.scoped( + Effect.gen(function* () { + const ledger = yield* makeAgentProcessLedger({ exitWait: "2 seconds" }); + const leader = spawn(process.execPath, ["-e", LEADER], { + detached: true, + stdio: ["ignore", "pipe", "ignore"], + }); + const exit = once(leader, "exit"); + const [line] = yield* Effect.promise(() => once(leader.stdout!, "data")); + const shellPid = Number(String(line).trim()); + yield* ledger.track(THREAD, { + pid: leader.pid!, + exited: Effect.promise(() => exit).pipe(Effect.asVoid), + }); + + yield* ledger.beforeStop(THREAD); + // The stop reaches the agent's own group; the shell lives on in its own. + process.kill(-leader.pid!, "SIGTERM"); + yield* Effect.promise(() => exit); + expect(isRunning(shellPid)).toBe(true); + + expect(yield* ledger.confirm(THREAD)).toEqual({ confirmed: true }); + expect(isRunning(shellPid)).toBe(false); + // Proven once, it is forgotten. + expect(yield* ledger.confirm(THREAD)).toEqual({ confirmed: true }); + }), + ), + ); + + it.live("says so when the agent's process can't be ended", () => + Effect.scoped( + Effect.gen(function* () { + const ledger = yield* makeAgentProcessLedger({ + exitWait: "50 millis", + settleWait: "50 millis", + probe: { + ...realProcessProbe, + exists: () => true, + signal: () => {}, + descendants: () => [], + }, + }); + yield* ledger.track(THREAD, { pid: 424242, exited: Effect.never }); + const confirmation = yield* ledger.confirm(THREAD); + expect(confirmation).toEqual({ + confirmed: false, + reason: "its process 424242 didn't exit", + }); + // Another thread's agents aren't its business. + expect(yield* ledger.confirm(ThreadId.make("thread-other"))).toEqual({ confirmed: true }); + }), + ), + ); +}); From eded41fafff76b2e41d7d67a2ae3d25198bd120f Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:56:59 -0500 Subject: [PATCH 13/48] feat(codex): a stopped app server, and the shells it started, are proven gone The runtime exposes the app server's pid and exit; the adapter tracks it in an agent-process ledger, notes what it started before closing it, and answers confirmThreadStopped from the ledger. --- .../src/provider/Layers/CodexAdapter.test.ts | 52 +++++++++++++++- .../src/provider/Layers/CodexAdapter.ts | 10 +++ .../provider/Layers/CodexSessionRuntime.ts | 7 +++ .../src/provider/agentProcessLedger.test.ts | 42 +++---------- .../server/src/provider/agentProcessLedger.ts | 21 +++++++ apps/server/src/provider/testAgentProcess.ts | 61 +++++++++++++++++++ 6 files changed, 158 insertions(+), 35 deletions(-) create mode 100644 apps/server/src/provider/testAgentProcess.ts diff --git a/apps/server/src/provider/Layers/CodexAdapter.test.ts b/apps/server/src/provider/Layers/CodexAdapter.test.ts index 318eca8d4..9b6bb768e 100644 --- a/apps/server/src/provider/Layers/CodexAdapter.test.ts +++ b/apps/server/src/provider/Layers/CodexAdapter.test.ts @@ -40,6 +40,12 @@ import type * as CodexClient from "effect-codex-app-server/client"; import type * as EffectCodexSchema from "effect-codex-app-server/schema"; import { attachmentRelativePath } from "../../attachmentStore.ts"; +import type { TrackedAgentProcess } from "../agentProcessLedger.ts"; +import { + isProcessRunning, + startTestAgentProcess, + type TestAgentProcess, +} from "../testAgentProcess.ts"; import { ServerConfig } from "../../config.ts"; import { ServerSettingsService } from "../../serverSettings.ts"; import { @@ -513,6 +519,8 @@ class FakeCodexRuntime implements CodexSessionRuntimeShape { close = Effect.promise(() => this.closeImpl()); + process?: TrackedAgentProcess; + emit(event: ProviderEvent) { return Queue.offer(this.eventQueue, event).pipe(Effect.asVoid); } @@ -534,7 +542,11 @@ function makeRuntimeFactory() { }; } -function makeScopedRuntimeFactory(options?: { readonly failConstruction?: boolean }) { +function makeScopedRuntimeFactory(options?: { + readonly failConstruction?: boolean; + /** Its app server: closing the runtime stops it, as the real one's scope does. */ + readonly agent?: TestAgentProcess; +}) { const runtimes: Array = []; const releasedThreadIds: Array = []; @@ -555,6 +567,11 @@ function makeScopedRuntimeFactory(options?: { readonly failConstruction?: boolea } const runtime = new FakeCodexRuntime(runtimeOptions); + const agent = options?.agent; + if (agent !== undefined) { + runtime.process = { pid: agent.pid, exited: agent.exited }; + runtime.closeImpl.mockImplementation(() => Effect.runPromise(agent.stop)); + } runtimes.push(runtime); return runtime; }), @@ -3368,6 +3385,39 @@ scopedLifecycleLayer("CodexAdapterLive scoped lifecycle", (it) => { ); }); +it.live.skipIf(process.platform === "win32")( + "proves the app server and the shells it started are gone after a stop", + () => + Effect.gen(function* () { + const agent = yield* startTestAgentProcess; + const factory = makeScopedRuntimeFactory({ agent }); + const adapter = yield* makeCodexAdapter(decodeCodexSettings({}), { + makeRuntime: factory.factory, + }); + const threadId = asThreadId("thread-confirm"); + yield* adapter.startSession({ + provider: ProviderDriverKind.make("codex"), + threadId, + runtimeMode: "full-access", + }); + yield* adapter.stopSession(threadId); + // The stop ended the app server; a shell it put in a group of its own lives on. + assert.equal(isProcessRunning(agent.shellPid), true); + + assert.deepStrictEqual(yield* adapter.confirmThreadStopped!(threadId), { confirmed: true }); + assert.equal(isProcessRunning(agent.shellPid), false); + }).pipe( + Effect.scoped, + Effect.provide( + Layer.mergeAll( + ServerConfig.layerTest(process.cwd(), process.cwd()), + ServerSettingsService.layerTest(), + providerSessionDirectoryTestLayer, + ).pipe(Layer.provideMerge(NodeServices.layer)), + ), + ), +); + const scopedFailureRuntimeFactory = makeScopedRuntimeFactory({ failConstruction: true }); const scopedFailureLayer = it.layer( Layer.effect( diff --git a/apps/server/src/provider/Layers/CodexAdapter.ts b/apps/server/src/provider/Layers/CodexAdapter.ts index 7e9462fd6..2f213bbf0 100644 --- a/apps/server/src/provider/Layers/CodexAdapter.ts +++ b/apps/server/src/provider/Layers/CodexAdapter.ts @@ -100,6 +100,7 @@ import { import { type EventNdjsonLogger, makeEventNdjsonLogger } from "./EventNdjsonLogger.ts"; import { initializeCodexAppServerClient, makeCodexAppServerClient } from "./CodexProvider.ts"; import { holdLaunchGate, type LaunchGate } from "../managedRuntime/LaunchGate.ts"; +import { makeAgentProcessLedger } from "../agentProcessLedger.ts"; import { mcpRoomEndpointUrl } from "../../mcp/McpHttpServer.ts"; import { roomSideKindOf, roomToolsFor } from "../../mcp/roomToolAccess.ts"; @@ -3249,6 +3250,8 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( options?.nativeEventLogger === undefined ? nativeEventLogger : undefined; const runtimeEventQueue = yield* Queue.unbounded(); const sessions = new Map(); + /** Each app-server it started, until proven gone (confirmThreadStopped). */ + const processes = yield* makeAgentProcessLedger(); /** * Threads whose message went again after a sign-in failure, until their * agent answers: a fresh app server that fails too means the sign-in itself @@ -3464,6 +3467,10 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( ), ); + if (runtime.process !== undefined) { + yield* processes.track(input.threadId, runtime.process); + } + const turnProgress: CodexTurnProgress = { workedTurnIds: new Set() }; const suggestions: CodexSuggestionState = { awaiting: new Map(), @@ -4543,6 +4550,8 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( } session.stopped = true; sessions.delete(session.threadId); + // What the app-server started, noted while it still runs (agentProcessLedger). + yield* processes.beforeStop(session.threadId); yield* session.runtime.close.pipe(Effect.ignore); yield* Effect.ignore(Scope.close(session.scope, Exit.void)); if (options?.interruptEventFiber !== false) { @@ -4624,6 +4633,7 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( listSessions, hasSession, stopAll, + confirmThreadStopped: processes.confirm, get streamEvents() { return Stream.fromQueue(runtimeEventQueue); }, diff --git a/apps/server/src/provider/Layers/CodexSessionRuntime.ts b/apps/server/src/provider/Layers/CodexSessionRuntime.ts index 62c4cf3d5..101879234 100644 --- a/apps/server/src/provider/Layers/CodexSessionRuntime.ts +++ b/apps/server/src/provider/Layers/CodexSessionRuntime.ts @@ -88,6 +88,7 @@ import { CODEX_PREVIEW_PANEL_DEVELOPER_INSTRUCTIONS, } from "../CodexDeveloperInstructions.ts"; import { FILE_LINK_INSTRUCTIONS } from "../fileLinkInstructions.ts"; +import { processExit, type TrackedAgentProcess } from "../agentProcessLedger.ts"; import { CODEX_BROWSER_TOKEN_ENV_VAR, codexAppServerArgs, @@ -431,6 +432,11 @@ export interface CodexSessionRuntimeShape { ) => Effect.Effect; readonly events: Stream.Stream; readonly close: Effect.Effect; + /** + * The app-server process, for proof it is gone after a stop + * (agentProcessLedger). Absent only for test doubles that run none. + */ + readonly process?: TrackedAgentProcess; } export type CodexThreadGoal = EffectCodexSchema.V2ThreadGoalSetResponse__ThreadGoal; @@ -3407,5 +3413,6 @@ const makeCodexSessionRuntimeIn = ( }), events: Stream.fromQueue(events), close, + process: { pid: child.pid, exited: processExit(child.pid, child.exitCode) }, } satisfies CodexSessionRuntimeShape; }); diff --git a/apps/server/src/provider/agentProcessLedger.test.ts b/apps/server/src/provider/agentProcessLedger.test.ts index a419bf7b8..f59f6dd9f 100644 --- a/apps/server/src/provider/agentProcessLedger.test.ts +++ b/apps/server/src/provider/agentProcessLedger.test.ts @@ -1,54 +1,29 @@ -import { spawn } from "node:child_process"; -import { once } from "node:events"; - import { ThreadId } from "@threadlines/contracts"; import { it } from "@effect/vitest"; import * as Effect from "effect/Effect"; import { describe, expect } from "vite-plus/test"; import { makeAgentProcessLedger, realProcessProbe } from "./agentProcessLedger.ts"; +import { isProcessRunning, startTestAgentProcess } from "./testAgentProcess.ts"; const THREAD = ThreadId.make("thread-ledger"); -// Starts a shell of its own (a process group it leads), prints its pid, and waits. -const LEADER = ` -const { spawn } = require("node:child_process"); -const shell = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { - detached: true, - stdio: "ignore", -}); -shell.unref(); -process.stdout.write(String(shell.pid) + "\\n"); -setInterval(() => {}, 1000); -`; - -const isRunning = (pid: number) => realProcessProbe.exists(pid); - describe.skipIf(process.platform === "win32")("agent process ledger", () => { it.live("ends what an agent left in a process group of its own, then confirms", () => Effect.scoped( Effect.gen(function* () { const ledger = yield* makeAgentProcessLedger({ exitWait: "2 seconds" }); - const leader = spawn(process.execPath, ["-e", LEADER], { - detached: true, - stdio: ["ignore", "pipe", "ignore"], - }); - const exit = once(leader, "exit"); - const [line] = yield* Effect.promise(() => once(leader.stdout!, "data")); - const shellPid = Number(String(line).trim()); - yield* ledger.track(THREAD, { - pid: leader.pid!, - exited: Effect.promise(() => exit).pipe(Effect.asVoid), - }); + const agent = yield* startTestAgentProcess; + yield* ledger.track(THREAD, { pid: agent.pid, exited: agent.exited }); yield* ledger.beforeStop(THREAD); // The stop reaches the agent's own group; the shell lives on in its own. - process.kill(-leader.pid!, "SIGTERM"); - yield* Effect.promise(() => exit); - expect(isRunning(shellPid)).toBe(true); + yield* agent.stop; + yield* agent.exited; + expect(isProcessRunning(agent.shellPid)).toBe(true); expect(yield* ledger.confirm(THREAD)).toEqual({ confirmed: true }); - expect(isRunning(shellPid)).toBe(false); + expect(isProcessRunning(agent.shellPid)).toBe(false); // Proven once, it is forgotten. expect(yield* ledger.confirm(THREAD)).toEqual({ confirmed: true }); }), @@ -69,8 +44,7 @@ describe.skipIf(process.platform === "win32")("agent process ledger", () => { }, }); yield* ledger.track(THREAD, { pid: 424242, exited: Effect.never }); - const confirmation = yield* ledger.confirm(THREAD); - expect(confirmation).toEqual({ + expect(yield* ledger.confirm(THREAD)).toEqual({ confirmed: false, reason: "its process 424242 didn't exit", }); diff --git a/apps/server/src/provider/agentProcessLedger.ts b/apps/server/src/provider/agentProcessLedger.ts index 768e6ed85..84a123a53 100644 --- a/apps/server/src/provider/agentProcessLedger.ts +++ b/apps/server/src/provider/agentProcessLedger.ts @@ -23,6 +23,7 @@ import type { ThreadId } from "@threadlines/contracts"; import * as Duration from "effect/Duration"; import * as Effect from "effect/Effect"; import * as Option from "effect/Option"; + import type * as Scope from "effect/Scope"; import type { ProviderStopConfirmation } from "./Services/ProviderAdapter.ts"; @@ -78,6 +79,24 @@ export const realProcessProbe: ProcessProbe = { }, }; +/** + * A spawned process's exit as an effect that can't fail: its handle's own + * exit, or, when that can't be read, its pid going away. + */ +export const processExit = ( + pid: number, + exitCode: Effect.Effect, + probe: ProcessProbe = realProcessProbe, +): Effect.Effect => + exitCode.pipe( + Effect.asVoid, + Effect.catchCause(() => + Effect.gen(function* () { + while (probe.exists(pid)) yield* Effect.sleep(Duration.millis(100)); + }), + ), + ); + export interface AgentProcessLedger { /** A process started for `sessionKey`. Forgotten once it and all it started are gone. */ readonly track: (sessionKey: ThreadId, agent: TrackedAgentProcess) => Effect.Effect; @@ -142,6 +161,8 @@ export const makeAgentProcessLedger = ( const track: AgentProcessLedger["track"] = (sessionKey, agent) => Effect.gen(function* () { + // Signalling pid 0 or below would reach this server's own group. + if (!Number.isInteger(agent.pid) || agent.pid <= 0) return; const entry: Entry = { sessionKey, pid: agent.pid, diff --git a/apps/server/src/provider/testAgentProcess.ts b/apps/server/src/provider/testAgentProcess.ts new file mode 100644 index 000000000..c2309141f --- /dev/null +++ b/apps/server/src/provider/testAgentProcess.ts @@ -0,0 +1,61 @@ +// @effect-diagnostics nodeBuiltinImport:off +/** + * For tests: a stand-in agent process that starts a shell in a process group + * of its own, the way Grok's background shells (and any `setsid` child) do, + * so a test can prove a driver's stop notes what its agent started and that + * `confirmThreadStopped` ends it. POSIX only. + */ +import { spawn } from "node:child_process"; +import { once } from "node:events"; + +import * as Effect from "effect/Effect"; +import type * as Scope from "effect/Scope"; + +import { realProcessProbe } from "./agentProcessLedger.ts"; + +const LEADER = ` +const { spawn } = require("node:child_process"); +const shell = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { + detached: true, + stdio: "ignore", +}); +shell.unref(); +process.stdout.write(String(shell.pid) + "\\n"); +setInterval(() => {}, 1000); +`; + +export interface TestAgentProcess { + readonly pid: number; + readonly exited: Effect.Effect; + /** The shell it started, in a process group of its own. */ + readonly shellPid: number; + /** What a driver's own stop does: signals the agent's process group, not the shell's. */ + readonly stop: Effect.Effect; +} + +export const isProcessRunning = (pid: number) => realProcessProbe.exists(pid); + +/** Starts one; whatever is left of it is killed when the scope closes. */ +export const startTestAgentProcess: Effect.Effect = + Effect.acquireRelease( + Effect.promise(async () => { + const leader = spawn(process.execPath, ["-e", LEADER], { + detached: true, + stdio: ["ignore", "pipe", "ignore"], + }); + const exit = once(leader, "exit"); + const [line] = await once(leader.stdout!, "data"); + const pid = leader.pid!; + return { + pid, + exited: Effect.promise(() => exit).pipe(Effect.asVoid), + shellPid: Number(String(line).trim()), + stop: Effect.sync(() => realProcessProbe.signal(-pid, "SIGTERM")), + }; + }), + (agent) => + Effect.sync(() => { + realProcessProbe.signal(-agent.pid, "SIGKILL"); + realProcessProbe.signal(-agent.shellPid, "SIGKILL"); + }), + ); From fd547addd7d16753c6b42f96c9ace6d920982d80 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 20:58:49 -0500 Subject: [PATCH 14/48] feat(acp): a stopped agent, and the shells it put in groups of their own, are proven gone The ACP runtime exposes the agent's pid and exit; the adapter tracks it, notes what the agent started before closing its scope (Grok's background shells live in process groups of their own and outlive the agent's group kill), and answers confirmThreadStopped from its ledger. --- .../src/provider/Layers/CodexAdapter.test.ts | 4 +- .../src/provider/acp/AcpAdapter.test.ts | 58 +++++++++++++++++++ apps/server/src/provider/acp/AcpAdapter.ts | 10 ++++ .../src/provider/acp/AcpSessionRuntime.ts | 10 ++++ 4 files changed, 81 insertions(+), 1 deletion(-) diff --git a/apps/server/src/provider/Layers/CodexAdapter.test.ts b/apps/server/src/provider/Layers/CodexAdapter.test.ts index 9b6bb768e..53ed5666c 100644 --- a/apps/server/src/provider/Layers/CodexAdapter.test.ts +++ b/apps/server/src/provider/Layers/CodexAdapter.test.ts @@ -570,7 +570,9 @@ function makeScopedRuntimeFactory(options?: { const agent = options?.agent; if (agent !== undefined) { runtime.process = { pid: agent.pid, exited: agent.exited }; - runtime.closeImpl.mockImplementation(() => Effect.runPromise(agent.stop)); + runtime.closeImpl.mockImplementation(() => + Effect.runPromise(agent.stop).then(() => undefined), + ); } runtimes.push(runtime); return runtime; diff --git a/apps/server/src/provider/acp/AcpAdapter.test.ts b/apps/server/src/provider/acp/AcpAdapter.test.ts index 136670263..375ee0ba8 100644 --- a/apps/server/src/provider/acp/AcpAdapter.test.ts +++ b/apps/server/src/provider/acp/AcpAdapter.test.ts @@ -28,6 +28,7 @@ import { import { ServerConfig } from "../../config.ts"; import { ServerSettingsService } from "../../serverSettings.ts"; +import { isProcessRunning } from "../testAgentProcess.ts"; import type { UsageLedgerEntry } from "../../usage/usageLedger.ts"; import { type AcpAdapterShape, acpDiffFileChanges, makeAcpAdapter } from "./AcpAdapter.ts"; import { discoverAcpModels } from "./AcpProvider.ts"; @@ -410,6 +411,63 @@ nativeControlsAdapterTestLayer("an agent driven through its own controls", (it) ); }); +it.live.skipIf(isWindows)( + "proves an agent, and a shell it left in a process group of its own, are gone after a stop", + () => + Effect.gen(function* () { + const dir = yield* Effect.promise(() => mkdtemp(path.join(os.tmpdir(), "acp-confirm-"))); + const shellPidPath = path.join(dir, "shell.pid"); + // A helper the agent runs in the background starts a shell in a group + // of its own, the way Grok's background shells do. + const helper = `const { spawn } = require("node:child_process"); const shell = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { detached: true, stdio: "ignore" }); shell.unref(); require("node:fs").writeFileSync(${JSON.stringify(shellPidPath)}, String(shell.pid)); setInterval(() => {}, 1000);`; + const wrapperPath = path.join(dir, "agent-with-shell.sh"); + yield* Effect.promise(async () => { + await writeFile( + wrapperPath, + `#!/bin/sh\n${JSON.stringify(nodeExe)} -e '${helper}' &\nexec ${JSON.stringify(nodeExe)} ${JSON.stringify(mockAgentPath)} "$@"\n`, + "utf8", + ); + await chmod(wrapperPath, 0o755); + }); + const settings = yield* ServerSettingsService; + yield* settings.updateSettings({ providers: { cursor: { binaryPath: wrapperPath } } }); + const adapter = yield* makeAcpAdapter(CURSOR_ACP_DESCRIPTOR, decodeCursorSettings({}), { + resolveSettings: yield* makeResolveCursorSettings, + }); + const threadId = ThreadId.make("cursor-confirm-stopped"); + yield* adapter.startSession({ + threadId, + provider: ProviderDriverKind.make("cursor"), + cwd: process.cwd(), + runtimeMode: "full-access", + modelSelection: { instanceId: ProviderInstanceId.make("cursor"), model: "default" }, + }); + const shellPid = Number(yield* Effect.promise(() => waitForFileContent(shellPidPath, 2_000))); + yield* Effect.addFinalizer(() => + Effect.sync(() => { + try { + process.kill(shellPid, "SIGKILL"); + } catch {} + }), + ); + + yield* adapter.stopSession(threadId); + // The stop ended the agent's own process group; the shell lives on. + assert.isTrue(isProcessRunning(shellPid)); + + assert.deepStrictEqual(yield* adapter.confirmThreadStopped!(threadId), { confirmed: true }); + assert.isFalse(isProcessRunning(shellPid)); + }).pipe( + Effect.scoped, + Effect.provide( + Layer.mergeAll( + ServerSettingsService.layerTest(), + ServerConfig.layerTest(process.cwd(), { prefix: "threadlines-acp-confirm-test-" }), + ).pipe(Layer.provideMerge(NodeServices.layer)), + ), + ), +); + cursorAdapterTestLayer("CursorAdapterLive", (it) => { it.effect("starts a session and maps mock ACP prompt flow to runtime events", () => Effect.gen(function* () { diff --git a/apps/server/src/provider/acp/AcpAdapter.ts b/apps/server/src/provider/acp/AcpAdapter.ts index 97363a8fd..c26c12ba3 100644 --- a/apps/server/src/provider/acp/AcpAdapter.ts +++ b/apps/server/src/provider/acp/AcpAdapter.ts @@ -26,6 +26,7 @@ import { TurnId, } from "@threadlines/contracts"; import { withFailureCategory } from "../providerFailureCategory.ts"; +import { makeAgentProcessLedger } from "../agentProcessLedger.ts"; import { isProviderPlanGateMessage, isProviderPlanGateReply, @@ -559,6 +560,8 @@ export function makeAcpAdapter( options?.nativeEventLogger === undefined ? nativeEventLogger : undefined; const sessions = new Map(); + /** Each agent process it started, until proven gone (confirmThreadStopped). */ + const processes = yield* makeAgentProcessLedger(); /** * Each agent session's turn history by its id, kept after the session * stops: a fork of it finds its boundary here. @@ -1008,6 +1011,9 @@ export function makeAcpAdapter( if (ctx.notificationFiber) { yield* Fiber.interrupt(ctx.notificationFiber); } + // What the agent started (Grok's shells live in process groups of + // their own), noted while it still runs (agentProcessLedger). + yield* processes.beforeStop(ctx.threadId); yield* Effect.ignore(Scope.close(ctx.scope, Exit.void)); sessions.delete(ctx.threadId); yield* offerRuntimeEvent({ @@ -1290,6 +1296,9 @@ export function makeAcpAdapter( }), ), ); + if (acp.process !== undefined) { + yield* processes.track(input.threadId, acp.process); + } /** * Shows a request from the agent in order with the output before it, @@ -2453,6 +2462,7 @@ export function makeAcpAdapter( listSessions, hasSession, stopAll, + confirmThreadStopped: processes.confirm, streamEvents: Stream.fromPubSub(runtimeEventPubSub), } satisfies AcpAdapterShape; }); diff --git a/apps/server/src/provider/acp/AcpSessionRuntime.ts b/apps/server/src/provider/acp/AcpSessionRuntime.ts index 917e548eb..0e5d8218e 100644 --- a/apps/server/src/provider/acp/AcpSessionRuntime.ts +++ b/apps/server/src/provider/acp/AcpSessionRuntime.ts @@ -17,6 +17,7 @@ import type * as EffectAcpProtocol from "effect-acp/protocol"; import { hideWindowsConsole, makeCommandWithShell } from "@threadlines/shared/childProcess"; import { planCliSpawn } from "../../cliSpawn.ts"; +import { processExit, type TrackedAgentProcess } from "../agentProcessLedger.ts"; import { collectSessionConfigOptionValues, extractModelConfigId, @@ -177,6 +178,11 @@ export interface AcpSessionRuntimeStartResult { } export interface AcpSessionRuntimeShape { + /** + * The agent's process, for proof it is gone after a stop + * (agentProcessLedger). Absent only for test doubles that run none. + */ + readonly process?: TrackedAgentProcess; readonly handleRequestPermission: EffectAcpClient.AcpClientShape["handleRequestPermission"]; readonly handleElicitation: EffectAcpClient.AcpClientShape["handleElicitation"]; readonly handleReadTextFile: EffectAcpClient.AcpClientShape["handleReadTextFile"]; @@ -1007,6 +1013,10 @@ const makeAcpSessionRuntime = ( request: (method, payload) => runLoggedRequest(method, payload, acp.raw.request(method, payload)), notify: acp.raw.notify, + process: { + pid: Number(child.pid), + exited: processExit(Number(child.pid), child.exitCode), + }, } satisfies AcpSessionRuntimeShape; }); From e99be4fb10d6ca3d55ad6e6a58a47efceeae8ab1 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:00:40 -0500 Subject: [PATCH 15/48] feat(pi, muse): a stopped Pi process or Muse host, and what it started, are proven gone PiRpc and MuseHost expose their process ids; each adapter tracks every process or host generation under the session key, notes what it started before it closes it (replacements included), and answers confirmThreadStopped from its ledger. Muse still sends task/stopAll first. --- .../src/provider/Layers/MuseAdapter.test.ts | 35 +++++++++++++++++-- .../server/src/provider/Layers/MuseAdapter.ts | 12 +++++++ apps/server/src/provider/Layers/PiAdapter.ts | 12 +++++++ apps/server/src/provider/muse/MuseHost.ts | 3 ++ apps/server/src/provider/pi/PiRpc.ts | 3 ++ 5 files changed, 62 insertions(+), 3 deletions(-) diff --git a/apps/server/src/provider/Layers/MuseAdapter.test.ts b/apps/server/src/provider/Layers/MuseAdapter.test.ts index 248d2540b..afc5e37b4 100644 --- a/apps/server/src/provider/Layers/MuseAdapter.test.ts +++ b/apps/server/src/provider/Layers/MuseAdapter.test.ts @@ -30,6 +30,11 @@ import * as TestClock from "effect/testing/TestClock"; import { describe, expect } from "vite-plus/test"; import { ServerConfig } from "../../config.ts"; +import { + isProcessRunning, + startTestAgentProcess, + type TestAgentProcess, +} from "../testAgentProcess.ts"; import { steerRejectionOf } from "../Errors.ts"; import { type MuseHost, @@ -64,7 +69,7 @@ interface FakeHost { type Responder = (call: Call) => Record | MuseHostError | undefined; /** A scripted host factory: each host the adapter starts is recorded in `hosts`. */ -function scriptedHosts(responder: Responder = () => undefined) { +function scriptedHosts(responder: Responder = () => undefined, agent?: TestAgentProcess) { const hosts: Array = []; let commandIds = 0; const startHost = (options: MuseHostOptions) => @@ -103,7 +108,12 @@ function scriptedHosts(responder: Responder = () => undefined) { return {}; } }; - yield* Effect.addFinalizer(() => Deferred.succeed(exited, { kind: "clean" })); + // With a real process behind it, closing the host ends that process first. + yield* Effect.addFinalizer(() => + (agent === undefined ? Effect.void : Effect.andThen(agent.stop, agent.exited)).pipe( + Effect.andThen(Deferred.succeed(exited, { kind: "clean" })), + ), + ); const host: MuseHost = { initialize: { serverVersion: "1.4.9", @@ -130,6 +140,7 @@ function scriptedHosts(responder: Responder = () => undefined) { exited: Deferred.await(exited), hasExited: Deferred.isDone(exited), stderrTail: () => "", + ...(agent !== undefined ? { pid: agent.pid } : {}), }; hosts.push({ options, @@ -156,10 +167,12 @@ const setup = ( input: { readonly responder?: Responder; readonly allowApiKeyBilling?: boolean; + /** A real process behind each host, for proof it is gone after a stop. */ + readonly agent?: TestAgentProcess; } = {}, ) => Effect.gen(function* () { - const scripted = scriptedHosts(input.responder); + const scripted = scriptedHosts(input.responder, input.agent); const adapter = yield* makeMuseAdapter({ instanceId: INSTANCE_ID, settings: { @@ -232,6 +245,22 @@ const shellApproval = (input: { }); describe("MuseAdapter", () => { + it.live.skipIf(process.platform === "win32")( + "proves a stopped host, and what it left in a process group of its own, are gone", + () => + Effect.gen(function* () { + const agent = yield* startTestAgentProcess; + const { adapter, threadId, start } = yield* setup({ agent }); + yield* start(); + yield* adapter.stopSession(threadId); + // The host is gone; a shell it put in a group of its own lives on. + expect(isProcessRunning(agent.shellPid)).toBe(true); + + expect(yield* adapter.confirmThreadStopped!(threadId)).toEqual({ confirmed: true }); + expect(isProcessRunning(agent.shellPid)).toBe(false); + }).pipe(Effect.scoped, Effect.provide(testLayer)), + ); + it.effect("sends the device note as a part of its own, outside the words Muse shows", () => Effect.gen(function* () { const { hosts, adapter, threadId, start } = yield* setup(); diff --git a/apps/server/src/provider/Layers/MuseAdapter.ts b/apps/server/src/provider/Layers/MuseAdapter.ts index 20e2ff56e..28fa32929 100644 --- a/apps/server/src/provider/Layers/MuseAdapter.ts +++ b/apps/server/src/provider/Layers/MuseAdapter.ts @@ -47,6 +47,7 @@ import { type ProviderFailureCategory, } from "@threadlines/contracts"; import { withFailureCategory } from "../providerFailureCategory.ts"; +import { makeAgentProcessLedger } from "../agentProcessLedger.ts"; import { MANAGED_WORKTREE_INSTRUCTION, renderThreadContextSeed, @@ -433,6 +434,8 @@ export function makeMuseAdapter(options: MuseAdapterOptions) { const boundInstanceId = options.instanceId; const runtimeEvents = yield* PubSub.unbounded(); const sessions = new Map(); + /** Each host it started, until proven gone (confirmThreadStopped). */ + const processes = yield* makeAgentProcessLedger(); const nowIso = Effect.map(DateTime.now, DateTime.formatIso); const stamp = Effect.all({ @@ -1869,6 +1872,12 @@ export function makeMuseAdapter(options: MuseAdapterOptions) { }), ), ); + if (host.pid !== undefined) { + yield* processes.track(ctx.threadId, { + pid: host.pid, + exited: Effect.asVoid(host.exited), + }); + } yield* checkBilling(host); const config = mcpConfig(ctx, host.initialize.grantedCapabilities); const permission = profile.permissionProfile @@ -2026,6 +2035,8 @@ export function makeMuseAdapter(options: MuseAdapterOptions) { if (!state) return; state.closing = true; ctx.host = undefined; + // What the host started, noted while it still runs (agentProcessLedger). + yield* processes.beforeStop(ctx.threadId); yield* Scope.close(state.scope, Exit.void); }); @@ -2884,6 +2895,7 @@ export function makeMuseAdapter(options: MuseAdapterOptions) { rollbackThread, readSubagentTranscript, stopAll, + confirmThreadStopped: processes.confirm, streamEvents: Stream.fromPubSub(runtimeEvents), } satisfies MuseAdapterShape; }); diff --git a/apps/server/src/provider/Layers/PiAdapter.ts b/apps/server/src/provider/Layers/PiAdapter.ts index 7d9fb2a42..9bcf79155 100644 --- a/apps/server/src/provider/Layers/PiAdapter.ts +++ b/apps/server/src/provider/Layers/PiAdapter.ts @@ -73,6 +73,7 @@ import { createHash } from "node:crypto"; import * as NodePath from "node:path"; import { resolveAttachmentPath } from "../../attachmentStore.ts"; +import { makeAgentProcessLedger } from "../agentProcessLedger.ts"; import { ServerConfig } from "../../config.ts"; import { BROWSER_MCP_SERVER_NAME, @@ -376,6 +377,8 @@ export function makePiAdapter(options: PiAdapterOptions) { const boundInstanceId = options.instanceId; const runtimeEvents = yield* PubSub.unbounded(); const sessions = new Map(); + /** Each Pi process it started, until proven gone (confirmThreadStopped). */ + const processes = yield* makeAgentProcessLedger(); const startRpc = options.startRpc ?? startPiRpc; const nowIso = Effect.map(DateTime.now, DateTime.formatIso); @@ -1316,6 +1319,8 @@ export function makePiAdapter(options: PiAdapterOptions) { if (!proc) return; proc.closing = true; ctx.proc = undefined; + // What Pi started, noted while it still runs (agentProcessLedger). + yield* processes.beforeStop(ctx.threadId); yield* Scope.close(proc.scope, Exit.void); }); @@ -1403,6 +1408,12 @@ export function makePiAdapter(options: PiAdapterOptions) { }), ), ); + if (rpc.pid !== undefined) { + yield* processes.track(ctx.threadId, { + pid: rpc.pid, + exited: Effect.asVoid(rpc.exited), + }); + } // Without its commands a marker would reach the model as text. const commands = decodePi(PiCommands)( yield* rpc @@ -2394,6 +2405,7 @@ export function makePiAdapter(options: PiAdapterOptions) { })), rollbackThread, stopAll, + confirmThreadStopped: processes.confirm, streamEvents: Stream.fromPubSub(runtimeEvents), } satisfies PiAdapterShape; }); diff --git a/apps/server/src/provider/muse/MuseHost.ts b/apps/server/src/provider/muse/MuseHost.ts index 4873eafb9..312dca5a2 100644 --- a/apps/server/src/provider/muse/MuseHost.ts +++ b/apps/server/src/provider/muse/MuseHost.ts @@ -159,6 +159,8 @@ export interface MuseHost { /** Whether the process has exited. */ readonly hasExited: Effect.Effect; readonly stderrTail: () => string; + /** The host's process id, for proof it is gone after a stop (agentProcessLedger). Absent for doubles. */ + readonly pid?: number; } export interface MuseHostOptions { @@ -461,5 +463,6 @@ export const startMuseHost = ( exited: Deferred.await(exited), hasExited: Deferred.isDone(exited), stderrTail: () => stderrTail, + pid: Number(child.pid), } satisfies MuseHost; }); diff --git a/apps/server/src/provider/pi/PiRpc.ts b/apps/server/src/provider/pi/PiRpc.ts index de6eb9f96..787978510 100644 --- a/apps/server/src/provider/pi/PiRpc.ts +++ b/apps/server/src/provider/pi/PiRpc.ts @@ -104,6 +104,8 @@ export interface PiRpc { readonly exited: Effect.Effect; readonly hasExited: Effect.Effect; readonly stderrTail: () => string; + /** Pi's process id, for proof it is gone after a stop (agentProcessLedger). Absent for doubles. */ + readonly pid?: number; } export interface PiRpcOptions { @@ -434,5 +436,6 @@ export const startPiRpc = ( exited: Deferred.await(exited), hasExited: Deferred.isDone(exited), stderrTail: () => stderrTail, + pid: Number(child.pid), } satisfies PiRpc; }); From 6b7d71328b549f11be153e1f769d45e65e3305c0 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:03:30 -0500 Subject: [PATCH 16/48] feat(claude): a stopped session's Claude Code process, and what it started, are proven gone The SDK reports the process it spawned (onProcessSpawned); the adapter tracks it, notes what it started before closing the query, watches the pid for its exit only after the stop (bounded), and answers confirmThreadStopped from its ledger. A session whose process the SDK never reported can't be proven stopped, and says so. --- .../src/provider/Layers/ClaudeAdapter.test.ts | 61 ++++++++++++++++ .../src/provider/Layers/ClaudeAdapter.ts | 69 ++++++++++++++++++- 2 files changed, 129 insertions(+), 1 deletion(-) diff --git a/apps/server/src/provider/Layers/ClaudeAdapter.test.ts b/apps/server/src/provider/Layers/ClaudeAdapter.test.ts index 91e19d2bf..c36b569a3 100644 --- a/apps/server/src/provider/Layers/ClaudeAdapter.test.ts +++ b/apps/server/src/provider/Layers/ClaudeAdapter.test.ts @@ -60,6 +60,7 @@ import * as TestClock from "effect/testing/TestClock"; import { attachmentRelativePath, createAttachmentId } from "../../attachmentStore.ts"; import { ServerConfig } from "../../config.ts"; +import { isProcessRunning, startTestAgentProcess } from "../testAgentProcess.ts"; import { ServerSettingsService } from "../../serverSettings.ts"; import { ProviderAdapterValidationError } from "../Errors.ts"; import { mcpSessionRegistry } from "../../mcp/McpSessionRegistry.ts"; @@ -6122,6 +6123,66 @@ describe("ClaudeAdapterLive", () => { ); }); + it.live.skipIf(process.platform === "win32")( + "proves a stopped session's Claude Code process, and what it started, are gone", + () => + Effect.gen(function* () { + const agent = yield* startTestAgentProcess; + const reported = new FakeClaudeQuery(); + (reported as { close: () => void }).close = () => { + reported.closeCalls += 1; + Effect.runSync(agent.stop); + }; + const layerFor = (createQuery: ClaudeAdapterLiveOptions["createQuery"]) => + Layer.effect( + ClaudeAdapter, + Effect.gen(function* () { + return yield* makeClaudeAdapter(decodeClaudeSettings({}), { + ...(createQuery !== undefined ? { createQuery } : {}), + }); + }), + ).pipe( + Layer.provideMerge(ServerConfig.layerTest("/tmp/claude-adapter-test", "/tmp")), + Layer.provideMerge(ServerSettingsService.layerTest()), + Layer.provideMerge(NodeServices.layer), + ); + const stopAndConfirm = Effect.gen(function* () { + const adapter = yield* ClaudeAdapter; + yield* adapter.startSession({ + threadId: THREAD_ID, + provider: ProviderDriverKind.make("claudeAgent"), + runtimeMode: "full-access", + }); + yield* adapter.stopSession(THREAD_ID); + return yield* adapter.confirmThreadStopped!(THREAD_ID); + }).pipe(Effect.provideService(Random.Random, makeDeterministicRandomService())); + + // The SDK says which process it started: the stop is proven, and a + // shell Claude Code left in a process group of its own is ended. + const confirmed = yield* stopAndConfirm.pipe( + Effect.provide( + layerFor((input) => { + (input.options as { onProcessSpawned?: (pid: number) => void }).onProcessSpawned?.( + agent.pid, + ); + return reported; + }), + ), + ); + assert.deepEqual(confirmed, { confirmed: true }); + assert.equal(isProcessRunning(agent.shellPid), false); + + // An SDK that never says can't prove anything. + const unproven = yield* stopAndConfirm.pipe( + Effect.provide(layerFor(() => new FakeClaudeQuery())), + ); + assert.deepEqual(unproven, { + confirmed: false, + reason: "Claude Code didn't report which process it started", + }); + }).pipe(Effect.scoped), + ); + it.effect("stopSession does not throw into the SDK prompt consumer", () => { // The SDK consumes user messages via `for await (... of prompt)`. // Stopping a session must end that loop cleanly — not throw an error. diff --git a/apps/server/src/provider/Layers/ClaudeAdapter.ts b/apps/server/src/provider/Layers/ClaudeAdapter.ts index c9008dc37..23871eafc 100644 --- a/apps/server/src/provider/Layers/ClaudeAdapter.ts +++ b/apps/server/src/provider/Layers/ClaudeAdapter.ts @@ -111,6 +111,7 @@ import { } from "../claudeLongRunningTool.ts"; import { CLAUDE_PREVIEW_PANEL_INSTRUCTIONS } from "../previewPanelInstructions.ts"; import { nonEmptyTrimmed } from "../providerSnapshot.ts"; +import { makeAgentProcessLedger, realProcessProbe } from "../agentProcessLedger.ts"; import { httpStatusFailureCategory, withFailureCategory } from "../providerFailureCategory.ts"; import { buildAgentPageInstructions, @@ -582,6 +583,12 @@ interface ClaudeSessionContext { session: ProviderSession; readonly promptQueue: Queue.Queue; readonly query: ClaudeQueryRuntime; + /** + * The Claude Code process the SDK started, as it reported it, with its + * exit: completed once the process is seen gone after a stop. Undefined + * when the SDK didn't say (a test double, or an SDK that dropped the hook). + */ + readonly cli: { readonly pid: number; readonly exited: Deferred.Deferred } | undefined; streamFiber: Fiber.Fiber | undefined; readonly startedAt: string; // The access level's permission mode on the current model; plan turns @@ -2783,6 +2790,9 @@ const CLAUDE_GOAL_REPLY_TIMEOUT = Duration.seconds(30); const CLAUDE_GOAL_WATCH_INTERVAL = Duration.millis(1_500); /** How long after the model proposes a goal its record may still land. */ const CLAUDE_GOAL_PROPOSAL_WATCH_MS = 60_000; +/** How long a stopped session's Claude Code process is watched for its exit, and how often. */ +const CLI_EXIT_WATCH_MS = 15_000; +const CLI_EXIT_POLL_MS = 100; /** How long "steer now" waits for the SDK to take the steer before it moves * running work to the background anyway. */ const CLAUDE_STEER_TAKEN_TIMEOUT = Duration.seconds(2); @@ -4227,6 +4237,26 @@ export const makeClaudeAdapter = Effect.fn("makeClaudeAdapter")(function* ( >(undefined)); const sessions = new Map(); + /** Each Claude Code process it started, until proven gone (confirmThreadStopped). */ + const processes = yield* makeAgentProcessLedger(); + /** Session keys whose process the SDK never reported: their stop can't be proven. */ + const unreportedProcesses = new Set(); + const adapterScope = yield* Effect.scope; + /** + * Waits (bounded) for a stopped session's Claude Code process to go: the + * SDK ends it on timers of its own (SIGTERM after 2 s, SIGKILL 5 s later) + * and gives no exit to await, so its pid is watched, only after the stop. + */ + const watchCliExit = (cli: NonNullable) => + Effect.gen(function* () { + for (let waited = 0; waited < CLI_EXIT_WATCH_MS; waited += CLI_EXIT_POLL_MS) { + if (!realProcessProbe.exists(cli.pid)) { + yield* Deferred.succeed(cli.exited, undefined); + return; + } + yield* Effect.sleep(Duration.millis(CLI_EXIT_POLL_MS)); + } + }).pipe(Effect.forkIn(adapterScope), Effect.asVoid); /** * Sessions Claude Code refused to start, by id. A thread keeps the id of * its session for the next start to resume, and one that never began left @@ -8357,6 +8387,8 @@ export const makeClaudeAdapter = Effect.fn("makeClaudeAdapter")(function* ( if (context.stopped) return; context.stopped = true; + // What Claude Code started, noted while it still runs (agentProcessLedger). + yield* processes.beforeStop(context.session.threadId); // First, so nothing this runtime left running can use its tools again. if (context.mcpCredentialGeneration !== undefined) { yield* mcpSessionRegistry.revoke(context.session.threadId, context.mcpCredentialGeneration); @@ -8453,6 +8485,7 @@ export const makeClaudeAdapter = Effect.fn("makeClaudeAdapter")(function* ( emitRuntimeError(context, "Failed to close Claude runtime query.", cause), ), ); + if (context.cli !== undefined) yield* watchCliExit(context.cli); const updatedAt = yield* nowIso; context.session = { @@ -9732,11 +9765,22 @@ export const makeClaudeAdapter = Effect.fn("makeClaudeAdapter")(function* ( "claude.query.path_to_executable": claudeBinaryPath, }); + // The SDK says which process it started (an option its types leave + // out), so a stop can be proven: docs/design/fleet.md, Part 3b. + let reportedPid: number | undefined; + const reportingOptions: ClaudeQueryOptions & { + readonly onProcessSpawned: (pid: number) => void; + } = { + ...queryOptions, + onProcessSpawned: (pid) => { + reportedPid = pid; + }, + }; const queryRuntime = yield* Effect.try({ try: () => createQuery({ prompt, - options: queryOptions, + options: reportingOptions, }), catch: (cause) => new ProviderAdapterProcessError({ @@ -9774,10 +9818,26 @@ export const makeClaudeAdapter = Effect.fn("makeClaudeAdapter")(function* ( updatedAt: startedAt, }; + const cli = + reportedPid === undefined + ? undefined + : { pid: reportedPid, exited: yield* Deferred.make() }; + if (cli === undefined) { + unreportedProcesses.add(threadId); + } else { + unreportedProcesses.delete(threadId); + yield* processes.track(threadId, { + pid: cli.pid, + exited: Deferred.await(cli.exited), + // The SDK spawns it in this server's own process group. + ownGroup: false, + }); + } const context: ClaudeSessionContext = { session, promptQueue, query: queryRuntime, + cli, streamFiber: undefined, startedAt, basePermissionMode: permissionMode, @@ -10915,6 +10975,13 @@ export const makeClaudeAdapter = Effect.fn("makeClaudeAdapter")(function* ( listSessions, hasSession, stopAll, + confirmThreadStopped: (threadId) => + [...unreportedProcesses].some((key) => parseSessionKey(key).threadId === threadId) + ? Effect.succeed({ + confirmed: false, + reason: "Claude Code didn't report which process it started", + } as const) + : processes.confirm(threadId), get streamEvents() { return Stream.fromQueue(runtimeEventQueue); }, From 3e971cc57fccef2a5f435e3f4d07395f85524ba2 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:05:00 -0500 Subject: [PATCH 17/48] feat(opencode): a stopped thread's OpenCode work is confirmed ended by the server that ran it One OpenCode server runs every thread's agent, so its process can't prove one thread stopped. A stop that interrupted work records the thread's OpenCode sessions and shells; confirmThreadStopped asks the server until none of them is active or running (a little while, for interrupts still settling), and says when the server doesn't answer. --- .../provider/Layers/OpenCodeAdapter.test.ts | 26 +++++ .../src/provider/Layers/OpenCodeAdapter.ts | 95 +++++++++++++++++++ 2 files changed, 121 insertions(+) diff --git a/apps/server/src/provider/Layers/OpenCodeAdapter.test.ts b/apps/server/src/provider/Layers/OpenCodeAdapter.test.ts index 4e9a83c94..3f351eb1e 100644 --- a/apps/server/src/provider/Layers/OpenCodeAdapter.test.ts +++ b/apps/server/src/provider/Layers/OpenCodeAdapter.test.ts @@ -332,6 +332,32 @@ describe("OpenCodeAdapter", () => { }).pipe(Effect.scoped, Effect.provide(testLayer)), ); + it.live("confirms a stopped thread only once OpenCode says its shells ended", () => + Effect.gen(function* () { + // The shared server ends the shell a moment after the stop removes it. + let shellReads = 0; + const { adapter, calls, send, next, threadId, cwd } = yield* harnessWith({ + "shell.get": () => ({ + location: {}, + data: { status: ++shellReads === 1 ? "running" : "killed" }, + }), + }); + yield* adapter.startSession({ threadId, cwd, runtimeMode: "full-access" }); + yield* adapter.sendTurn({ threadId, messageId: MessageId.make("m1"), input: "serve" }); + yield* send("session.execution.started", { sessionID: ROOT }); + yield* backgroundShell(send, "sh_1"); + yield* next((event) => event.type === "task.started"); + + yield* adapter.stopSession(threadId); + expect(yield* adapter.confirmThreadStopped!(threadId)).toEqual({ confirmed: true }); + expect(shellReads).toBe(2); + expect(calls.some((call) => call.method === "session.active")).toBe(true); + // Proven once, nothing is left to ask about. + expect(yield* adapter.confirmThreadStopped!(threadId)).toEqual({ confirmed: true }); + expect(shellReads).toBe(2); + }).pipe(Effect.scoped, Effect.provide(testLayer)), + ); + it.live("stops one background shell, and its end is reported once", () => Effect.gen(function* () { const { adapter, calls, send, next, threadId, cwd } = yield* harness; diff --git a/apps/server/src/provider/Layers/OpenCodeAdapter.ts b/apps/server/src/provider/Layers/OpenCodeAdapter.ts index 44df34e87..912df02ab 100644 --- a/apps/server/src/provider/Layers/OpenCodeAdapter.ts +++ b/apps/server/src/provider/Layers/OpenCodeAdapter.ts @@ -97,6 +97,8 @@ import { type OpenCodeClient, type OpenCodeError, type OpenCodeModelRef, + openCodeErrorDetail, + openCodeErrorStatus, openCodeErrorTag, openCodeModelSlug, parseOpenCodeModelSlug, @@ -151,6 +153,10 @@ const BUILD_AGENT = "build"; const PLAN_AGENT = "plan"; /** How long a stop may take before the adapter checks whether OpenCode is still running. */ const STOP_SETTLE_TIMEOUT = Duration.seconds(10); +/** Asking OpenCode whether stopped work ended: each call's limit, and how often to ask again. */ +const OPENCODE_CONFIRM_CALL_TIMEOUT = Duration.seconds(5); +const OPENCODE_CONFIRM_RETRY = Duration.millis(500); +const OPENCODE_CONFIRM_ATTEMPTS = 10; /** How long a turn waits for the execution OpenCode starts to deliver steered input. */ const HELD_END_GRACE = Duration.seconds(5); /** Lists (models, agents) are empty for a moment in a directory the server has not loaded yet. */ @@ -413,6 +419,20 @@ export function makeOpenCodeAdapter(options: OpenCodeAdapterOptions) { const runtimeEvents = yield* PubSub.unbounded(); const sessions = new Map(); + /** + * Work a stop interrupted, by session key, until OpenCode confirms it + * ended (confirmThreadStopped). One OpenCode server runs every thread's + * agent, so the server's own word is the proof: its process stays up. + */ + const stoppedWork = new Map< + ThreadId, + { + readonly client: OpenCodeClient; + readonly directory: string; + readonly sessionIds: ReadonlyArray; + readonly shellIds: ReadonlyArray; + } + >(); /** Every OpenCode session we own, root or child, to the thread it belongs to. */ const owners = new Map(); /** Which session registered each Threadlines tool server, so a stale stop never removes a newer one. */ @@ -3048,6 +3068,14 @@ export function makeOpenCodeAdapter(options: OpenCodeAdapterOptions) { .filter((child) => child.status === "running") .map((child) => child.sessionId), ]; + if (running.length > 0 || ctx.backgroundShells.size > 0) { + stoppedWork.set(threadId, { + client: ctx.client, + directory: ctx.directory, + sessionIds: running, + shellIds: [...ctx.backgroundShells.keys()], + }); + } yield* Effect.forEach( running, (sessionId) => @@ -3147,6 +3175,72 @@ export function makeOpenCodeAdapter(options: OpenCodeAdapterOptions) { const stopAll: OpenCodeAdapterShape["stopAll"] = () => Effect.forEach([...sessions.keys()], stopSession, { discard: true }); + /** What OpenCode still runs of the work a stop interrupted: problems, in words. */ + const workLeft = (work: NonNullable>) => + Effect.gen(function* () { + const active = yield* runOpenCode("session.active", (signal) => + work.client.session.active({ signal }), + ).pipe(Effect.timeout(OPENCODE_CONFIRM_CALL_TIMEOUT)); + const runningShells = yield* Effect.forEach( + work.shellIds, + (id) => + runOpenCode("shell.get", (signal) => + work.client.shell.get({ id, location: { directory: work.directory } }, { signal }), + ).pipe( + Effect.map((shell) => (shell.data.status === "running" ? [id] : [])), + // Removed: gone. + Effect.catchIf( + (error) => openCodeErrorStatus(error) === 404, + () => Effect.succeed([]), + ), + Effect.timeout(OPENCODE_CONFIRM_CALL_TIMEOUT), + ), + { concurrency: "unbounded" }, + ).pipe(Effect.map((ids) => ids.flat())); + const busy = work.sessionIds.filter((id) => Object.hasOwn(active, id)); + return [ + ...(busy.length > 0 ? ["OpenCode still runs its work"] : []), + ...(runningShells.length > 0 + ? [ + `OpenCode still runs ${runningShells.length} shell${runningShells.length === 1 ? "" : "s"} it started`, + ] + : []), + ]; + }).pipe( + Effect.catch((error) => + Effect.succeed([`OpenCode's server didn't answer (${openCodeErrorDetail(error)})`]), + ), + ); + + /** + * Asks OpenCode whether the work a thread's stops interrupted has ended, + * waiting a little for interrupts still settling. A thread that was idle + * when it stopped left nothing to ask about. + */ + const confirmThreadStopped: NonNullable = ( + threadId, + ) => + Effect.gen(function* () { + const keys = [...stoppedWork.keys()].filter( + (key) => parseSessionKey(key).threadId === threadId, + ); + const problems: string[] = []; + for (const key of keys) { + const work = stoppedWork.get(key); + if (work === undefined) continue; + let left = yield* workLeft(work); + for (let attempt = 1; left.length > 0 && attempt < OPENCODE_CONFIRM_ATTEMPTS; attempt++) { + yield* Effect.sleep(OPENCODE_CONFIRM_RETRY); + left = yield* workLeft(work); + } + if (left.length === 0) stoppedWork.delete(key); + problems.push(...left); + } + return problems.length === 0 + ? ({ confirmed: true } as const) + : ({ confirmed: false, reason: problems.join("; ") } as const); + }); + yield* Effect.addFinalizer(() => stopAll().pipe(Effect.ignore, Effect.andThen(PubSub.shutdown(runtimeEvents))), ); @@ -3190,6 +3284,7 @@ export function makeOpenCodeAdapter(options: OpenCodeAdapterOptions) { deleteThread, readSubagentTranscript, stopAll, + confirmThreadStopped, streamEvents: Stream.fromPubSub(runtimeEvents), } satisfies OpenCodeAdapterShape; }); From 5d8f78fd047835ef02694e8c8af881e789b7bb77 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:07:24 -0500 Subject: [PATCH 18/48] test(claude): pass the query factory straight through in the confirmed-stop test --- apps/server/src/provider/Layers/ClaudeAdapter.test.ts | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/apps/server/src/provider/Layers/ClaudeAdapter.test.ts b/apps/server/src/provider/Layers/ClaudeAdapter.test.ts index c36b569a3..882263264 100644 --- a/apps/server/src/provider/Layers/ClaudeAdapter.test.ts +++ b/apps/server/src/provider/Layers/ClaudeAdapter.test.ts @@ -6133,13 +6133,11 @@ describe("ClaudeAdapterLive", () => { reported.closeCalls += 1; Effect.runSync(agent.stop); }; - const layerFor = (createQuery: ClaudeAdapterLiveOptions["createQuery"]) => + const layerFor = (createQuery: NonNullable) => Layer.effect( ClaudeAdapter, Effect.gen(function* () { - return yield* makeClaudeAdapter(decodeClaudeSettings({}), { - ...(createQuery !== undefined ? { createQuery } : {}), - }); + return yield* makeClaudeAdapter(decodeClaudeSettings({}), { createQuery }); }), ).pipe( Layer.provideMerge(ServerConfig.layerTest("/tmp/claude-adapter-test", "/tmp")), From e325c98d7f4ea2445879fd9fe7d444298ab4cbbd Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:27:23 -0500 Subject: [PATCH 19/48] perf(server): the process table is read only for a stop that's for a move, and off the event loop Every provider stop ran a synchronous ps. Now ProviderService asks the driver to note what an agent started (noteBeforeStop) only when the session fence holds the thread, which the projection already shows before the reactor stops its sessions; the drivers no longer read the table on their own stops. The read is an async execFile with a 2 second limit, and a table that couldn't be read makes the stop unconfirmed rather than confirmed blind. The Windows tree kill is async with the same limit. The move queue no longer sends a status that changes nothing: the engine refuses a command that records no events. --- apps/server/src/moves/MoveQueue.test.ts | 16 +++- apps/server/src/moves/MoveQueue.ts | 32 +++++--- .../src/orchestration/threadMoveDecisions.ts | 36 +++++---- .../src/provider/Layers/ClaudeAdapter.test.ts | 1 + .../src/provider/Layers/ClaudeAdapter.ts | 3 +- .../src/provider/Layers/CodexAdapter.test.ts | 2 + .../src/provider/Layers/CodexAdapter.ts | 3 +- .../src/provider/Layers/MuseAdapter.test.ts | 2 + .../server/src/provider/Layers/MuseAdapter.ts | 3 +- apps/server/src/provider/Layers/PiAdapter.ts | 3 +- .../provider/Layers/ProviderService.test.ts | 17 +++- .../src/provider/Layers/ProviderService.ts | 9 +++ .../src/provider/Services/ProviderAdapter.ts | 9 +++ .../src/provider/acp/AcpAdapter.test.ts | 2 + apps/server/src/provider/acp/AcpAdapter.ts | 5 +- .../src/provider/agentProcessLedger.test.ts | 2 +- .../server/src/provider/agentProcessLedger.ts | 77 ++++++++++++------- packages/shared/src/processTree.ts | 28 ++++++- 18 files changed, 182 insertions(+), 68 deletions(-) diff --git a/apps/server/src/moves/MoveQueue.test.ts b/apps/server/src/moves/MoveQueue.test.ts index 8d78fec4e..b413d9f75 100644 --- a/apps/server/src/moves/MoveQueue.test.ts +++ b/apps/server/src/moves/MoveQueue.test.ts @@ -57,6 +57,8 @@ async function harness(options?: { readonly model?: OrchestrationReadModel }) { quiesceCalls: 0, starts: 0, cancels: 0, + /** Commands the queue sent that recorded nothing (the real engine refuses them). */ + emptyCommands: 0, /** Runs right before a start, as if a provider turn slipped in. */ beforeStart: null as null | (() => Promise), }; @@ -66,9 +68,15 @@ async function harness(options?: { readonly model?: OrchestrationReadModel }) { const decided = yield* decideOrchestrationCommand({ command, readModel: model }).pipe( Effect.mapError((error) => new ThreadMoveError({ detail: error.message })), ); - for (const planned of Array.isArray(decided) ? decided : [decided]) { + const planned = Array.isArray(decided) ? decided : [decided]; + // As the real engine: a command that records nothing is refused. + if (planned.length === 0) { + state.emptyCommands += 1; + return yield* new ThreadMoveError({ detail: "Command produced no events." }); + } + for (const next of planned) { sequence += 1; - const event = { ...planned, sequence } as OrchestrationEvent; + const event = { ...next, sequence } as OrchestrationEvent; model = yield* projectEvent(model, event).pipe(Effect.orDie); yield* PubSub.publish(events, event); if (event.type === "thread.session-stop-requested" && state.stopsLand) { @@ -257,6 +265,8 @@ describe("MoveQueue", () => { expect(h.thread().pendingMove ?? null).toBeNull(); expect(h.thread().move?.queuedRequestId).toBeDefined(); expect(h.state.quiesceCalls).toBeGreaterThan(0); + // It never sent a command that would record nothing. + expect(h.state.emptyCommands).toBe(0); await h.close(); }); @@ -326,6 +336,8 @@ describe("MoveQueue", () => { h.state.online = true; await h.linkChanged(); await waitFor(() => (h.thread().move ?? null) !== null, "the move to start on reconnect"); + // Retrying the same hold sent nothing the engine would refuse. + expect(h.state.emptyCommands).toBe(0); await h.close(); }); diff --git a/apps/server/src/moves/MoveQueue.ts b/apps/server/src/moves/MoveQueue.ts index 50b839ea9..00034cb42 100644 --- a/apps/server/src/moves/MoveQueue.ts +++ b/apps/server/src/moves/MoveQueue.ts @@ -35,6 +35,7 @@ import { type ThreadMoveRetryInput, type ThreadMoveUnqueueInput, type ThreadMoveWait, + type ThreadPendingMove, } from "@threadlines/contracts"; import * as Context from "effect/Context"; import * as DateTime from "effect/DateTime"; @@ -52,6 +53,7 @@ import { ProjectionSnapshotQuery } from "../orchestration/Services/ProjectionSna import { firstMoveBlock, moveWaitsOf, + sameMoveStatus, type ThreadMoveRefusalEntry, threadMoveRefusals, } from "../orchestration/threadMoveDecisions.ts"; @@ -154,13 +156,19 @@ export const makeMoveQueue = ( const commandId = (requestId: string, step: string) => CommandId.make(`server:thread-move-queue:${requestId}:${step}:${randomUUID()}`); + /** + * Says what the move waits for and what holds it, when that changed: the + * engine refuses a command that records nothing. + */ const report = ( threadId: ThreadId, - requestId: ThreadMoveRequestId, + pending: ThreadPendingMove, waitsFor: ReadonlyArray, hold: ThreadMoveHold | null, ) => Effect.gen(function* () { + if (sameMoveStatus(pending, waitsFor, hold)) return; + const requestId = pending.requestId; yield* dispatchIgnoringRefusal({ type: "thread.move.queue-status", commandId: commandId(requestId, "status"), @@ -296,18 +304,18 @@ export const makeMoveQueue = ( /** Reports what it waits for, or ends it when something only the user can clear stands in the way. */ const settleRefusals = ( threadId: ThreadId, - requestId: ThreadMoveRequestId, + pending: ThreadPendingMove, refusals: ReadonlyArray, ) => Effect.gen(function* () { const block = firstMoveBlock(refusals); if (block !== null) { const gone = block.code === "archived" || block.code === "deleted"; - yield* end(threadId, requestId, gone ? "gone" : "blocked", block.words); + yield* end(threadId, pending.requestId, gone ? "gone" : "blocked", block.words); return true; } if (refusals.length > 0) { - yield* report(threadId, requestId, moveWaitsOf(refusals), null); + yield* report(threadId, pending, moveWaitsOf(refusals), null); return true; } return false; @@ -322,13 +330,11 @@ export const makeMoveQueue = ( yield* track(threadId, pending.requestId); const requestId = pending.requestId; // From the thread alone first: while it works, nothing else needs reading. - if (yield* settleRefusals(threadId, requestId, localRefusals(thread))) return; + if (yield* settleRefusals(threadId, pending, localRefusals(thread))) return; const model = yield* deps.readModel; const current = model.threads.find((candidate) => candidate.id === threadId) ?? thread; - if (yield* settleRefusals(threadId, requestId, threadMoveRefusals(model, current))) return; - if ((pending.waitsFor ?? []).length > 0) { - yield* report(threadId, requestId, [], pending.hold ?? null); - } + if (yield* settleRefusals(threadId, pending, threadMoveRefusals(model, current))) return; + yield* report(threadId, pending, [], pending.hold ?? null); // The barrier: the fence goes up and its sessions stop. if (pending.stage === "waiting" || !sessionStopped(thread)) { @@ -353,7 +359,9 @@ export const makeMoveQueue = ( // Every driver proves its agent's processes are gone. const confirmation = yield* deps.quiesce(threadId); if (!confirmation.confirmed) { - yield* report(threadId, requestId, [], { + const latest = (yield* deps.readThread(threadId))?.pendingMove ?? null; + if (latest === null || latest.requestId !== requestId) return; + yield* report(threadId, latest, [], { kind: "stop_unconfirmed", words: `Couldn't confirm the agent stopped: ${confirmation.reason}.`, }); @@ -365,7 +373,7 @@ export const makeMoveQueue = ( entry.again = true; return; } - if (yield* settleRefusals(threadId, requestId, localRefusals(after))) return; + if (yield* settleRefusals(threadId, after.pendingMove, localRefusals(after))) return; if (after.pendingMove.stage !== "quiescing" || !sessionStopped(after)) { // The stop is still landing in the projection; its event looks again. return yield* schedule(threadId, retryDelay(0)); @@ -381,7 +389,7 @@ export const makeMoveQueue = ( yield* forget(threadId); return; case "unreachable": - yield* report(threadId, requestId, [], { + yield* report(threadId, after.pendingMove, [], { kind: "target_unreachable", words: outcome.words, }); diff --git a/apps/server/src/orchestration/threadMoveDecisions.ts b/apps/server/src/orchestration/threadMoveDecisions.ts index 2f8da2ad2..b33bef264 100644 --- a/apps/server/src/orchestration/threadMoveDecisions.ts +++ b/apps/server/src/orchestration/threadMoveDecisions.ts @@ -18,10 +18,12 @@ import { type OrchestrationReadModel, type OrchestrationThread, type ThreadId, + type ThreadMoveHold, ThreadMoveNotStartedActivityKind, type ThreadMoveNotStartedPayload, type ThreadMoveWait, type ThreadMoveWaitCode, + type ThreadPendingMove, } from "@threadlines/contracts"; import { threadPullRequestAutomationArmed } from "@threadlines/shared/linkedPullRequests"; import { @@ -652,11 +654,27 @@ export function decideMoveQuiesce( ]; } -const sameWaits = (left: ReadonlyArray, right: ReadonlyArray) => - left.length === right.length && - left.every( - (entry, index) => entry.code === right[index]?.code && entry.words === right[index].words, +/** + * Whether a queued move already says it waits for `waitsFor` and is held by + * `hold`: a status that would change nothing isn't sent or recorded. + */ +export function sameMoveStatus( + pending: ThreadPendingMove, + waitsFor: ReadonlyArray, + hold: ThreadMoveHold | null, +): boolean { + const shown = pending.waitsFor ?? []; + const held = pending.hold ?? null; + return ( + shown.length === waitsFor.length && + shown.every( + (entry, index) => + entry.code === waitsFor[index]?.code && entry.words === waitsFor[index].words, + ) && + held?.kind === hold?.kind && + held?.words === hold?.words ); +} /** What the queued move waits for now, and what holds it; unchanged is no news. */ export function decideMoveQueueStatus( @@ -665,15 +683,7 @@ export function decideMoveQueueStatus( baseFor: EventBase, ): ThreadMoveDecision { const pending = currentPendingMove(thread, command.requestId); - if (pending === null) return []; - const hold = pending.hold ?? null; - if ( - sameWaits(pending.waitsFor ?? [], command.waitsFor) && - hold?.kind === command.hold?.kind && - hold?.words === command.hold?.words - ) { - return []; - } + if (pending === null || sameMoveStatus(pending, command.waitsFor, command.hold)) return []; return [ { ...baseFor(thread.id), diff --git a/apps/server/src/provider/Layers/ClaudeAdapter.test.ts b/apps/server/src/provider/Layers/ClaudeAdapter.test.ts index 882263264..b60b56306 100644 --- a/apps/server/src/provider/Layers/ClaudeAdapter.test.ts +++ b/apps/server/src/provider/Layers/ClaudeAdapter.test.ts @@ -6151,6 +6151,7 @@ describe("ClaudeAdapterLive", () => { provider: ProviderDriverKind.make("claudeAgent"), runtimeMode: "full-access", }); + yield* adapter.noteBeforeStop!(THREAD_ID); yield* adapter.stopSession(THREAD_ID); return yield* adapter.confirmThreadStopped!(THREAD_ID); }).pipe(Effect.provideService(Random.Random, makeDeterministicRandomService())); diff --git a/apps/server/src/provider/Layers/ClaudeAdapter.ts b/apps/server/src/provider/Layers/ClaudeAdapter.ts index 23871eafc..e03fd4923 100644 --- a/apps/server/src/provider/Layers/ClaudeAdapter.ts +++ b/apps/server/src/provider/Layers/ClaudeAdapter.ts @@ -8387,8 +8387,6 @@ export const makeClaudeAdapter = Effect.fn("makeClaudeAdapter")(function* ( if (context.stopped) return; context.stopped = true; - // What Claude Code started, noted while it still runs (agentProcessLedger). - yield* processes.beforeStop(context.session.threadId); // First, so nothing this runtime left running can use its tools again. if (context.mcpCredentialGeneration !== undefined) { yield* mcpSessionRegistry.revoke(context.session.threadId, context.mcpCredentialGeneration); @@ -10975,6 +10973,7 @@ export const makeClaudeAdapter = Effect.fn("makeClaudeAdapter")(function* ( listSessions, hasSession, stopAll, + noteBeforeStop: processes.beforeStop, confirmThreadStopped: (threadId) => [...unreportedProcesses].some((key) => parseSessionKey(key).threadId === threadId) ? Effect.succeed({ diff --git a/apps/server/src/provider/Layers/CodexAdapter.test.ts b/apps/server/src/provider/Layers/CodexAdapter.test.ts index 53ed5666c..d30d9dd81 100644 --- a/apps/server/src/provider/Layers/CodexAdapter.test.ts +++ b/apps/server/src/provider/Layers/CodexAdapter.test.ts @@ -3402,6 +3402,8 @@ it.live.skipIf(process.platform === "win32")( threadId, runtimeMode: "full-access", }); + // A stop for a move notes what it started first (ProviderService does). + yield* adapter.noteBeforeStop!(threadId); yield* adapter.stopSession(threadId); // The stop ended the app server; a shell it put in a group of its own lives on. assert.equal(isProcessRunning(agent.shellPid), true); diff --git a/apps/server/src/provider/Layers/CodexAdapter.ts b/apps/server/src/provider/Layers/CodexAdapter.ts index 2f213bbf0..aa575ed20 100644 --- a/apps/server/src/provider/Layers/CodexAdapter.ts +++ b/apps/server/src/provider/Layers/CodexAdapter.ts @@ -4550,8 +4550,6 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( } session.stopped = true; sessions.delete(session.threadId); - // What the app-server started, noted while it still runs (agentProcessLedger). - yield* processes.beforeStop(session.threadId); yield* session.runtime.close.pipe(Effect.ignore); yield* Effect.ignore(Scope.close(session.scope, Exit.void)); if (options?.interruptEventFiber !== false) { @@ -4633,6 +4631,7 @@ export const makeCodexAdapter = Effect.fn("makeCodexAdapter")(function* ( listSessions, hasSession, stopAll, + noteBeforeStop: processes.beforeStop, confirmThreadStopped: processes.confirm, get streamEvents() { return Stream.fromQueue(runtimeEventQueue); diff --git a/apps/server/src/provider/Layers/MuseAdapter.test.ts b/apps/server/src/provider/Layers/MuseAdapter.test.ts index afc5e37b4..7fcea081d 100644 --- a/apps/server/src/provider/Layers/MuseAdapter.test.ts +++ b/apps/server/src/provider/Layers/MuseAdapter.test.ts @@ -252,6 +252,8 @@ describe("MuseAdapter", () => { const agent = yield* startTestAgentProcess; const { adapter, threadId, start } = yield* setup({ agent }); yield* start(); + // A stop for a move notes what the host started first (ProviderService does). + yield* adapter.noteBeforeStop!(threadId); yield* adapter.stopSession(threadId); // The host is gone; a shell it put in a group of its own lives on. expect(isProcessRunning(agent.shellPid)).toBe(true); diff --git a/apps/server/src/provider/Layers/MuseAdapter.ts b/apps/server/src/provider/Layers/MuseAdapter.ts index 28fa32929..144d6a9ad 100644 --- a/apps/server/src/provider/Layers/MuseAdapter.ts +++ b/apps/server/src/provider/Layers/MuseAdapter.ts @@ -2035,8 +2035,6 @@ export function makeMuseAdapter(options: MuseAdapterOptions) { if (!state) return; state.closing = true; ctx.host = undefined; - // What the host started, noted while it still runs (agentProcessLedger). - yield* processes.beforeStop(ctx.threadId); yield* Scope.close(state.scope, Exit.void); }); @@ -2895,6 +2893,7 @@ export function makeMuseAdapter(options: MuseAdapterOptions) { rollbackThread, readSubagentTranscript, stopAll, + noteBeforeStop: processes.beforeStop, confirmThreadStopped: processes.confirm, streamEvents: Stream.fromPubSub(runtimeEvents), } satisfies MuseAdapterShape; diff --git a/apps/server/src/provider/Layers/PiAdapter.ts b/apps/server/src/provider/Layers/PiAdapter.ts index 9bcf79155..203be2a27 100644 --- a/apps/server/src/provider/Layers/PiAdapter.ts +++ b/apps/server/src/provider/Layers/PiAdapter.ts @@ -1319,8 +1319,6 @@ export function makePiAdapter(options: PiAdapterOptions) { if (!proc) return; proc.closing = true; ctx.proc = undefined; - // What Pi started, noted while it still runs (agentProcessLedger). - yield* processes.beforeStop(ctx.threadId); yield* Scope.close(proc.scope, Exit.void); }); @@ -2405,6 +2403,7 @@ export function makePiAdapter(options: PiAdapterOptions) { })), rollbackThread, stopAll, + noteBeforeStop: processes.beforeStop, confirmThreadStopped: processes.confirm, streamEvents: Stream.fromPubSub(runtimeEvents), } satisfies PiAdapterShape; diff --git a/apps/server/src/provider/Layers/ProviderService.test.ts b/apps/server/src/provider/Layers/ProviderService.test.ts index 278cf9836..ac01ac1fd 100644 --- a/apps/server/src/provider/Layers/ProviderService.test.ts +++ b/apps/server/src/provider/Layers/ProviderService.test.ts @@ -3065,12 +3065,13 @@ function makeFencedProviderServiceLayer() { const codex = makeFakeCodexAdapter(); const claude = makeFakeCodexAdapter(CLAUDE_AGENT_DRIVER); const fenced = new Set(); + const noteBeforeStop = vi.fn((_sessionKey: ThreadId): Effect.Effect => Effect.void); const confirmThreadStopped = vi.fn( (_threadId: ThreadId): Effect.Effect => Effect.succeed({ confirmed: true }), ); const registry = makeAdapterRegistryMock({ - [ProviderDriverKind.make("codex")]: { ...codex.adapter, confirmThreadStopped }, + [ProviderDriverKind.make("codex")]: { ...codex.adapter, confirmThreadStopped, noteBeforeStop }, // A driver that can't prove its stop. [ProviderDriverKind.make("claudeAgent")]: claude.adapter, }); @@ -3100,7 +3101,7 @@ function makeFencedProviderServiceLayer() { NodeServices.layer, ), ); - return { codex, claude, fenced, confirmThreadStopped, layer }; + return { codex, claude, fenced, confirmThreadStopped, noteBeforeStop, layer }; } const fence = makeFencedProviderServiceLayer(); @@ -3157,11 +3158,23 @@ fence.layer("ProviderServiceLive moving threads", (it) => { yield* startOn(codexInstanceId, MOVING); yield* startOn(codexInstanceId, participantSessionKey(MOVING, ROOM_AGENT)); yield* startOn(codexInstanceId, other); + // The move's fence is up before its sessions stop. + fence.fenced.add(MOVING); + fence.noteBeforeStop.mockClear(); assert.deepEqual(yield* provider.quiesceThread(MOVING), { confirmed: true }); const left = (yield* provider.listSessions()).map((session) => session.threadId); assert.deepEqual(left, [other]); assert.deepEqual(fence.confirmThreadStopped.mock.calls.at(-1), [MOVING]); + // What each stopping runtime started was noted first; a stop that isn't + // for a move never reads the process table. + assert.deepEqual( + fence.noteBeforeStop.mock.calls.map(([key]) => key).toSorted(), + [MOVING, participantSessionKey(MOVING, ROOM_AGENT)].toSorted(), + ); + yield* provider.stopSession({ threadId: other }); + assert.equal(fence.noteBeforeStop.mock.calls.length, 2); + fence.fenced.delete(MOVING); // A driver that can't prove its stop never passes for a thread it ran. yield* startOn(ProviderInstanceId.make("claudeAgent"), MOVING); diff --git a/apps/server/src/provider/Layers/ProviderService.ts b/apps/server/src/provider/Layers/ProviderService.ts index 8d80b7662..47cc09065 100644 --- a/apps/server/src/provider/Layers/ProviderService.ts +++ b/apps/server/src/provider/Layers/ProviderService.ts @@ -2587,6 +2587,15 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( lastResumeCursor = (yield* routed.adapter.listSessions()).find( (session) => session.threadId === routed.threadId, )?.resumeCursor; + // A thread the session fence holds is stopping for a move: what its + // agent started is noted first, so the move can prove it gone + // (quiesceThread). Every other stop skips the process table. + if ( + routed.adapter.noteBeforeStop !== undefined && + (yield* admission.refusal(parseSessionKey(routed.threadId).threadId)) !== null + ) { + yield* routed.adapter.noteBeforeStop(routed.threadId); + } yield* routed.adapter.stopSession(routed.threadId); } // A side answer's runtime is disposable: nothing ever resumes it, so diff --git a/apps/server/src/provider/Services/ProviderAdapter.ts b/apps/server/src/provider/Services/ProviderAdapter.ts index 36895d0d5..7b7c6e3af 100644 --- a/apps/server/src/provider/Services/ProviderAdapter.ts +++ b/apps/server/src/provider/Services/ProviderAdapter.ts @@ -469,6 +469,15 @@ export interface ProviderAdapterShape { */ readonly confirmThreadStopped?: (threadId: ThreadId) => Effect.Effect; + /** + * One session key's runtime is about to stop for a move: note what its + * processes started, so `confirmThreadStopped` can find what outlives the + * stop (an agent's shell in a process group of its own). ProviderService + * calls it only for a thread the session fence holds, so other stops pay + * nothing. Reads the process table off the event loop, bounded. + */ + readonly noteBeforeStop?: (sessionKey: ThreadId) => Effect.Effect; + /** * Canonical runtime event stream emitted by this adapter. */ diff --git a/apps/server/src/provider/acp/AcpAdapter.test.ts b/apps/server/src/provider/acp/AcpAdapter.test.ts index 375ee0ba8..ca4140439 100644 --- a/apps/server/src/provider/acp/AcpAdapter.test.ts +++ b/apps/server/src/provider/acp/AcpAdapter.test.ts @@ -451,6 +451,8 @@ it.live.skipIf(isWindows)( }), ); + // A stop for a move notes what it started first (ProviderService does). + yield* adapter.noteBeforeStop!(threadId); yield* adapter.stopSession(threadId); // The stop ended the agent's own process group; the shell lives on. assert.isTrue(isProcessRunning(shellPid)); diff --git a/apps/server/src/provider/acp/AcpAdapter.ts b/apps/server/src/provider/acp/AcpAdapter.ts index c26c12ba3..d4cce2b63 100644 --- a/apps/server/src/provider/acp/AcpAdapter.ts +++ b/apps/server/src/provider/acp/AcpAdapter.ts @@ -1011,9 +1011,7 @@ export function makeAcpAdapter( if (ctx.notificationFiber) { yield* Fiber.interrupt(ctx.notificationFiber); } - // What the agent started (Grok's shells live in process groups of - // their own), noted while it still runs (agentProcessLedger). - yield* processes.beforeStop(ctx.threadId); + yield* Effect.ignore(Scope.close(ctx.scope, Exit.void)); sessions.delete(ctx.threadId); yield* offerRuntimeEvent({ @@ -2462,6 +2460,7 @@ export function makeAcpAdapter( listSessions, hasSession, stopAll, + noteBeforeStop: processes.beforeStop, confirmThreadStopped: processes.confirm, streamEvents: Stream.fromPubSub(runtimeEventPubSub), } satisfies AcpAdapterShape; diff --git a/apps/server/src/provider/agentProcessLedger.test.ts b/apps/server/src/provider/agentProcessLedger.test.ts index f59f6dd9f..ed4e3792f 100644 --- a/apps/server/src/provider/agentProcessLedger.test.ts +++ b/apps/server/src/provider/agentProcessLedger.test.ts @@ -40,7 +40,7 @@ describe.skipIf(process.platform === "win32")("agent process ledger", () => { ...realProcessProbe, exists: () => true, signal: () => {}, - descendants: () => [], + descendants: () => Effect.succeed([]), }, }); yield* ledger.track(THREAD, { pid: 424242, exited: Effect.never }); diff --git a/apps/server/src/provider/agentProcessLedger.ts b/apps/server/src/provider/agentProcessLedger.ts index 84a123a53..573f65ed7 100644 --- a/apps/server/src/provider/agentProcessLedger.ts +++ b/apps/server/src/provider/agentProcessLedger.ts @@ -15,9 +15,9 @@ * exited after its whole tree was killed (`taskkill /T`), which can miss a * process that left the tree. */ -import { spawnSync } from "node:child_process"; +import { execFile } from "node:child_process"; -import { listDescendantProcessIds } from "@threadlines/shared/processTree"; +import { listDescendantProcessIdsAsync } from "@threadlines/shared/processTree"; import { parseSessionKey } from "@threadlines/shared/threadParticipants"; import type { ThreadId } from "@threadlines/contracts"; import * as Duration from "effect/Duration"; @@ -46,12 +46,18 @@ export interface ProcessProbe { /** The process (or, with a negative pid, any process of the group) still exists. */ readonly exists: (pid: number) => boolean; readonly signal: (pid: number, signal: NodeJS.Signals) => void; - /** Every live descendant, from one snapshot of the process table. */ - readonly descendants: (pid: number) => ReadonlyArray; + /** + * Every live descendant, from one snapshot of the process table, read off + * the event loop. Null when the table couldn't be read in time. + */ + readonly descendants: (pid: number) => Effect.Effect | null>; /** Windows: kills the process and its tree. */ - readonly killTree: (pid: number) => void; + readonly killTree: (pid: number) => Effect.Effect; } +/** How long one look at the process table, or one tree kill, may take. */ +const PROCESS_TABLE_TIMEOUT_MS = 2_000; + export const realProcessProbe: ProcessProbe = { platform: process.platform, exists: (pid) => { @@ -70,13 +76,20 @@ export const realProcessProbe: ProcessProbe = { // Already gone. } }, - descendants: listDescendantProcessIds, - killTree: (pid) => { - spawnSync("taskkill", ["/pid", String(pid), "/T", "/F"], { - stdio: "ignore", - windowsHide: true, - }); - }, + descendants: (pid) => + Effect.promise(() => listDescendantProcessIdsAsync(pid, PROCESS_TABLE_TIMEOUT_MS)), + killTree: (pid) => + Effect.promise( + () => + new Promise((resolve) => { + execFile( + "taskkill", + ["/pid", String(pid), "/T", "/F"], + { timeout: PROCESS_TABLE_TIMEOUT_MS, windowsHide: true }, + () => resolve(), + ); + }), + ), }; /** @@ -101,9 +114,11 @@ export interface AgentProcessLedger { /** A process started for `sessionKey`. Forgotten once it and all it started are gone. */ readonly track: (sessionKey: ThreadId, agent: TrackedAgentProcess) => Effect.Effect; /** - * The driver is about to stop `sessionKey`'s runtime: note what its - * processes started, while they can still be told apart from anyone - * else's (once a process dies its children belong to pid 1). + * `sessionKey`'s runtime is about to stop for a move (ProviderService asks, + * only for a thread its session fence holds): note what its processes + * started, while they can still be told apart from anyone else's (once a + * process dies its children belong to pid 1). Reads the process table off + * the event loop, for at most 2 seconds. */ readonly beforeStop: (sessionKey: ThreadId) => Effect.Effect; /** Everything tracked for `threadId`'s session keys is gone, or why that can't be said. */ @@ -130,8 +145,11 @@ interface Entry { readonly exited: Effect.Effect; readonly ownGroup: boolean; leaderGone: boolean; - /** What it had started when its stop began, and when that was noted. */ - noted: ReadonlyArray; + /** + * What it had started when its stop began, and when that was noted. Null: + * the process table couldn't be read then, so what it left can't be known. + */ + noted: ReadonlyArray | null; notedAt: number; } @@ -155,7 +173,7 @@ export const makeAgentProcessLedger = ( posix && entry.ownGroup ? probe.exists(-entry.pid) : !entry.leaderGone && probe.exists(entry.pid); - const notedAlive = (entry: Entry) => entry.noted.filter((pid) => probe.exists(pid)); + const notedAlive = (entry: Entry) => (entry.noted ?? []).filter((pid) => probe.exists(pid)); const settled = (entry: Entry) => entry.leaderGone && !groupAlive(entry) && notedAlive(entry).length === 0; @@ -186,13 +204,17 @@ export const makeAgentProcessLedger = ( }); const beforeStop: AgentProcessLedger["beforeStop"] = (sessionKey) => - Effect.sync(() => { - for (const entry of entries) { - if (entry.sessionKey !== sessionKey || entry.leaderGone) continue; - entry.noted = probe.descendants(entry.pid); - entry.notedAt = Date.now(); - } - }); + Effect.forEach( + [...entries].filter((entry) => entry.sessionKey === sessionKey && !entry.leaderGone), + (entry) => + probe.descendants(entry.pid).pipe( + Effect.map((noted) => { + entry.noted = noted; + entry.notedAt = Date.now(); + }), + ), + { discard: true, concurrency: "unbounded" }, + ); /** Waits for one tracked process and what it left; the problems that remain. */ const settle = (entry: Entry) => @@ -201,7 +223,7 @@ export const makeAgentProcessLedger = ( if (Option.isNone(yield* entry.exited.pipe(Effect.timeoutOption(exitWait)))) { if (posix && entry.ownGroup) probe.signal(-entry.pid, "SIGKILL"); else if (posix) probe.signal(entry.pid, "SIGKILL"); - else probe.killTree(entry.pid); + else yield* probe.killTree(entry.pid); if (Option.isNone(yield* entry.exited.pipe(Effect.timeoutOption(settleWait)))) { problems.push(`its process ${entry.pid} didn't exit`); return problems; @@ -224,6 +246,9 @@ export const makeAgentProcessLedger = ( if (groupAlive(entry)) { problems.push(`processes in its process group ${entry.pid} are still running`); } + if (entry.noted === null) { + problems.push("the process list couldn't be read when it stopped"); + } const left = notedAlive(entry); if (left.length > 0) { problems.push( diff --git a/packages/shared/src/processTree.ts b/packages/shared/src/processTree.ts index 45c8ca9e8..40d5a25a1 100644 --- a/packages/shared/src/processTree.ts +++ b/packages/shared/src/processTree.ts @@ -1,5 +1,5 @@ // @effect-diagnostics nodeBuiltinImport:off -import { execFileSync } from "node:child_process"; +import { execFile, execFileSync } from "node:child_process"; /** * Live descendant pids of a process (children, grandchildren, …) from a @@ -19,7 +19,33 @@ export function listDescendantProcessIds(rootPid: number): number[] { } catch { return []; } + return descendantsFromTable(table, rootPid); +} + +/** + * listDescendantProcessIds without blocking the event loop: `ps` runs as a + * child process, given at most `timeoutMs`. Null when the table couldn't be + * read (it timed out or failed); empty on Windows, which has no `ps`. + */ +export function listDescendantProcessIdsAsync( + rootPid: number, + timeoutMs: number, +): Promise { + if (process.platform === "win32") return Promise.resolve([]); + return new Promise((resolve) => { + execFile( + "ps", + ["-eo", "pid=,ppid="], + { encoding: "utf8", timeout: timeoutMs, maxBuffer: 16 * 1024 * 1024 }, + (error, stdout) => { + resolve(error === null ? descendantsFromTable(stdout, rootPid) : null); + }, + ); + }); +} +/** The descendants of `rootPid` in a `ps -eo pid=,ppid=` table. */ +export function descendantsFromTable(table: string, rootPid: number): number[] { const childrenByParent = new Map(); for (const line of table.split("\n")) { const match = /^\s*(\d+)\s+(\d+)\s*$/.exec(line); From 9104702d3a6a46f6e18d25780e5c9a991dc73d20 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:31:57 -0500 Subject: [PATCH 20/48] fix(server): an early start the move ends has what its agent started noted first quiesceThread ends early starts that are still starting; when one's agent already runs, each driver notes what it started before the stop, as a listed session's stop does. --- apps/server/src/provider/Layers/ProviderService.ts | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/apps/server/src/provider/Layers/ProviderService.ts b/apps/server/src/provider/Layers/ProviderService.ts index 47cc09065..c703e8228 100644 --- a/apps/server/src/provider/Layers/ProviderService.ts +++ b/apps/server/src/provider/Layers/ProviderService.ts @@ -3024,10 +3024,18 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( ).pipe(Effect.map((perAdapter) => perAdapter.flat())); const live = yield* liveOfThread; const ran = new Set(live.map((entry) => entry.instanceId)); - // Early starts still starting have no session to list yet. + // Early starts still starting have no session to list yet. One whose + // agent already runs has what it started noted before it is ended; a + // listed session's stop notes its own (stopSessionCore). const keys = new Set(live.map((entry) => entry.key)); - for (const key of earlyStartRuns.keys()) { - if (ofThread(key)) keys.add(ThreadId.make(key)); + for (const rawKey of earlyStartRuns.keys()) { + if (!ofThread(rawKey)) continue; + const key = ThreadId.make(rawKey); + keys.add(key); + if (live.some((entry) => entry.key === key)) continue; + for (const [, adapter] of yield* getAdapterEntries) { + if (adapter.noteBeforeStop !== undefined) yield* adapter.noteBeforeStop(key); + } } const problems: string[] = []; for (const key of keys) { From 947cb07aa53a30a7caa3c8364fa87bcdff20209d Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:31:57 -0500 Subject: [PATCH 21/48] perf(server): a queued move reads only the threads its checks need One look of the move queue loaded the whole command read model when the thread looked free: 566 ms with 3,000 threads of 30 messages, 859 ms with 10,000. It now reads the thread, its attached children and the threads in its checkout (threadMoveView.ts): 8.6 ms and 9.3 ms. ThreadMoves' preview, start and queue checks use the same read, and the startup scan lists only threads with a queued move. --- apps/server/src/moves/MoveQueue.test.ts | 9 ++- apps/server/src/moves/MoveQueue.ts | 36 +++++++--- apps/server/src/moves/ThreadMoves.ts | 19 +++-- apps/server/src/moves/threadMoveView.test.ts | 70 +++++++++++++++++++ apps/server/src/moves/threadMoveView.ts | 69 ++++++++++++++++++ .../src/orchestration/threadMoveDecisions.ts | 9 ++- 6 files changed, 191 insertions(+), 21 deletions(-) create mode 100644 apps/server/src/moves/threadMoveView.test.ts create mode 100644 apps/server/src/moves/threadMoveView.ts diff --git a/apps/server/src/moves/MoveQueue.test.ts b/apps/server/src/moves/MoveQueue.test.ts index b413d9f75..f8bb5b30d 100644 --- a/apps/server/src/moves/MoveQueue.test.ts +++ b/apps/server/src/moves/MoveQueue.test.ts @@ -143,7 +143,14 @@ async function harness(options?: { readonly model?: OrchestrationReadModel }) { linkChanges: Stream.fromPubSub(linkChanges), readThread: (threadId) => Effect.sync(() => model.threads.find((entry) => entry.id === threadId) ?? null), - readModel: Effect.sync(() => model), + readContext: (threadId) => + Effect.sync(() => { + const thread = model.threads.find((entry) => entry.id === threadId); + return thread === undefined ? null : { thread, view: model }; + }), + pendingThreadIds: Effect.sync(() => + model.threads.filter((thread) => (thread.pendingMove ?? null) !== null).map((t) => t.id), + ), quiesce: () => Effect.sync(() => { state.quiesceCalls += 1; diff --git a/apps/server/src/moves/MoveQueue.ts b/apps/server/src/moves/MoveQueue.ts index 00034cb42..156686a65 100644 --- a/apps/server/src/moves/MoveQueue.ts +++ b/apps/server/src/moves/MoveQueue.ts @@ -23,7 +23,6 @@ import { CommandId, type OrchestrationCommand, type OrchestrationEvent, - type OrchestrationReadModel, type OrchestrationThread, type ThreadId, ThreadMoveError, @@ -46,6 +45,7 @@ import * as Layer from "effect/Layer"; import * as Option from "effect/Option"; import * as Scope from "effect/Scope"; import * as Stream from "effect/Stream"; +import * as SqlClient from "effect/sql/SqlClient"; import { ComputerLinks } from "../links/ComputerLinks.ts"; import { OrchestrationEngineService } from "../orchestration/Services/OrchestrationEngine.ts"; @@ -60,6 +60,7 @@ import { import type { ProviderStopConfirmation } from "../provider/Services/ProviderAdapter.ts"; import { ProviderService } from "../provider/Services/ProviderService.ts"; import { type QueuedMoveStart, ThreadMoves, type ThreadMovesShape } from "./ThreadMoves.ts"; +import { makeThreadMoveContextReader, type ThreadMoveContext } from "./threadMoveView.ts"; export interface MoveQueueShape { readonly queue: ( @@ -86,7 +87,10 @@ export interface MoveQueueDeps { readonly linkChanges: Stream.Stream; /** The thread, archived ones too; null when it's gone. A read that fails is a defect: the look is tried again. */ readonly readThread: (threadId: ThreadId) => Effect.Effect; - readonly readModel: Effect.Effect; + /** What the move checks read about a thread (threadMoveView.ts); null when it's gone. */ + readonly readContext: (threadId: ThreadId) => Effect.Effect; + /** Threads with a queued move, for the startup scan. */ + readonly pendingThreadIds: Effect.Effect>; readonly quiesce: (threadId: ThreadId) => Effect.Effect; readonly checkQueue: ThreadMovesShape["checkQueue"]; readonly startQueued: ThreadMovesShape["startQueued"]; @@ -331,9 +335,10 @@ export const makeMoveQueue = ( const requestId = pending.requestId; // From the thread alone first: while it works, nothing else needs reading. if (yield* settleRefusals(threadId, pending, localRefusals(thread))) return; - const model = yield* deps.readModel; - const current = model.threads.find((candidate) => candidate.id === threadId) ?? thread; - if (yield* settleRefusals(threadId, pending, threadMoveRefusals(model, current))) return; + const context = yield* deps.readContext(threadId); + if (context === null) return yield* forget(threadId); + const refusals = threadMoveRefusals(context.view, context.thread); + if (yield* settleRefusals(threadId, pending, refusals)) return; yield* report(threadId, pending, [], pending.hold ?? null); // The barrier: the fence goes up and its sessions stop. @@ -472,12 +477,11 @@ export const makeMoveQueue = ( const events = yield* deps.subscribeEvents; // The moves the last process left, read off the startup path. yield* Effect.gen(function* () { - const model = yield* deps.readModel; - for (const thread of model.threads) { - const pending = thread.pendingMove ?? null; + for (const threadId of yield* deps.pendingThreadIds) { + const pending = (yield* deps.readThread(threadId))?.pendingMove ?? null; if (pending !== null) { - yield* track(thread.id, pending.requestId); - yield* schedule(thread.id, Duration.zero); + yield* track(threadId, pending.requestId); + yield* schedule(threadId, Duration.zero); } } }).pipe( @@ -547,6 +551,8 @@ export const MoveQueueLive = Layer.effect( const providers = yield* ProviderService; const threadMoves = yield* ThreadMoves; const links = yield* ComputerLinks; + const sql = yield* SqlClient.SqlClient; + const readContext = yield* makeThreadMoveContextReader; return yield* makeMoveQueue({ dispatch: engine.dispatch, subscribeEvents: engine.subscribeDomainEvents, @@ -555,7 +561,15 @@ export const MoveQueueLive = Layer.effect( snapshots .getThreadDetailById(threadId, { includeArchived: true }) .pipe(Effect.map(Option.getOrNull), Effect.orDie), - readModel: snapshots.getCommandReadModel().pipe(Effect.orDie), + readContext: (threadId) => readContext(threadId).pipe(Effect.orDie), + pendingThreadIds: sql<{ readonly threadId: ThreadId }>` + SELECT thread_id AS "threadId" + FROM projection_threads + WHERE pending_move_json IS NOT NULL AND deleted_at IS NULL + `.pipe( + Effect.map((rows) => rows.map((row) => row.threadId)), + Effect.orDie, + ), quiesce: providers.quiesceThread, checkQueue: threadMoves.checkQueue, startQueued: threadMoves.startQueued, diff --git a/apps/server/src/moves/ThreadMoves.ts b/apps/server/src/moves/ThreadMoves.ts index 75edbe4bf..18a824d86 100644 --- a/apps/server/src/moves/ThreadMoves.ts +++ b/apps/server/src/moves/ThreadMoves.ts @@ -59,6 +59,7 @@ import { threadMoveRefusals, } from "../orchestration/threadMoveDecisions.ts"; import { ProviderService } from "../provider/Services/ProviderService.ts"; +import { makeThreadMoveContextReader } from "./threadMoveView.ts"; import { OrchestrationEngineService } from "../orchestration/Services/OrchestrationEngine.ts"; import { ProjectionSnapshotQuery } from "../orchestration/Services/ProjectionSnapshotQuery.ts"; import { ProjectionThreadRepository } from "../persistence/Services/ProjectionThreads.ts"; @@ -162,6 +163,7 @@ const make = Effect.gen(function* () { const moveDir = (transferId: string) => NodePath.join(config.stateDir, "moves", transferId); + const readMoveContext = yield* makeThreadMoveContextReader; const readModel = snapshots .getCommandReadModel() .pipe(Effect.mapError(() => moveError("This computer's threads couldn't be read."))); @@ -248,11 +250,15 @@ const make = Effect.gen(function* () { */ const inspect = (threadId: ThreadId, forQueued?: ThreadMoveRequestId) => Effect.gen(function* () { - const model = yield* readModel; - const thread = model.threads.find((entry) => entry.id === threadId); - if (thread === undefined) return yield* moveError("That thread doesn't exist."); - const project = model.projects.find((entry) => entry.id === thread.projectId); - const refusals = threadMoveRefusals(model, thread); + // Only what the checks read: the whole read model costs about half a + // second on a big install (threadMoveView.ts). + const context = yield* readMoveContext(threadId).pipe( + Effect.mapError(() => moveError("This computer's threads couldn't be read.")), + ); + if (context === null) return yield* moveError("That thread doesn't exist."); + const { thread, view } = context; + const project = view.projects.find((entry) => entry.id === thread.projectId); + const refusals = threadMoveRefusals(view, thread); const pending = thread.pendingMove ?? null; let refusal: string | null = firstMoveBlock(refusals)?.words ?? @@ -288,10 +294,9 @@ const make = Effect.gen(function* () { ); const waitsFor = moveWaitsOf(refusals); return { - model, thread, // A move that doesn't wait refuses a busy thread too. - busy: threadMoveRefusal(model, thread), + busy: threadMoveRefusal(view, thread), preview: { ...(refusal !== null ? { refusal } : {}), ...(waitsFor.length > 0 ? { waitsFor } : {}), diff --git a/apps/server/src/moves/threadMoveView.test.ts b/apps/server/src/moves/threadMoveView.test.ts new file mode 100644 index 000000000..e9eaec659 --- /dev/null +++ b/apps/server/src/moves/threadMoveView.test.ts @@ -0,0 +1,70 @@ +import { ThreadId } from "@threadlines/contracts"; +import { assert, it } from "@effect/vitest"; +import * as NodeServices from "@effect/platform-node/NodeServices"; +import * as Effect from "effect/Effect"; +import * as Layer from "effect/Layer"; +import * as SqlClient from "effect/sql/SqlClient"; + +import { SqlitePersistenceMemory } from "../persistence/Layers/Sqlite.ts"; +import { RepositoryIdentityResolverLive } from "../project/Layers/RepositoryIdentityResolver.ts"; +import { OrchestrationProjectionSnapshotQueryLive } from "../orchestration/Layers/ProjectionSnapshotQuery.ts"; +import { threadMoveRefusals } from "../orchestration/threadMoveDecisions.ts"; +import { makeThreadMoveContextReader } from "./threadMoveView.ts"; + +const layer = it.layer( + OrchestrationProjectionSnapshotQueryLive.pipe( + Layer.provideMerge(RepositoryIdentityResolverLive), + Layer.provideMerge(SqlitePersistenceMemory), + Layer.provideMerge(NodeServices.layer), + ), +); + +layer("threadMoveView", (it) => { + it.effect( + "reads the thread, its attached children and the threads in its checkout, and nothing else", + () => + Effect.gen(function* () { + const sql = yield* SqlClient.SqlClient; + yield* sql`INSERT INTO projection_projects (project_id, title, workspace_root, default_model_selection_json, scripts_json, created_at, updated_at, deleted_at) + VALUES ('project-1', 'Project', '/repo', '{"provider":"codex","model":"gpt-5-codex"}', '[]', '2026-02-24T00:00:00.000Z', '2026-02-24T00:00:00.000Z', NULL)`; + const thread = ( + id: string, + worktree: string | null, + parent: string | null = null, + deletedAt: string | null = null, + ) => + sql`INSERT INTO projection_threads (thread_id, project_id, title, model_selection_json, runtime_mode, interaction_mode, branch, worktree_path, latest_turn_id, latest_user_message_at, pending_approval_count, pending_user_input_count, has_actionable_proposed_plan, parent_thread_id, attached_to_parent, created_at, updated_at, deleted_at) + VALUES (${id}, 'project-1', ${id}, '{"provider":"codex","model":"gpt-5-codex"}', 'full-access', 'default', NULL, ${worktree}, NULL, NULL, 0, 0, 0, ${parent}, ${parent === null ? 0 : 1}, '2026-02-24T00:00:00.000Z', '2026-02-24T00:00:00.000Z', ${deletedAt})`; + yield* thread("moving", "/worktrees/a"); + yield* thread("child", "/worktrees/child", "moving"); + yield* thread("same-folder", "/worktrees/a"); + yield* thread("gone-same-folder", "/worktrees/a", null, "2026-02-25T00:00:00.000Z"); + yield* thread("project-folder", null); + yield* thread("elsewhere", "/worktrees/b"); + + const read = yield* makeThreadMoveContextReader; + const context = yield* read(ThreadId.make("moving")); + assert.isNotNull(context); + assert.deepEqual(context!.view.threads.map((entry) => entry.id).toSorted(), [ + "child", + "moving", + "same-folder", + ]); + assert.deepEqual( + context!.view.projects.map((project) => project.id), + ["project-1"], + ); + assert.include( + threadMoveRefusals(context!.view, context!.thread).map((entry) => entry.code), + "has_children", + ); + // A thread in the project folder is matched by that folder, and no other thread works there. + const inProject = yield* read(ThreadId.make("project-folder")); + assert.deepEqual( + inProject!.view.threads.map((entry) => entry.id), + ["project-folder"], + ); + assert.isNull(yield* read(ThreadId.make("never-was"))); + }), + ); +}); diff --git a/apps/server/src/moves/threadMoveView.ts b/apps/server/src/moves/threadMoveView.ts new file mode 100644 index 000000000..c193cf474 --- /dev/null +++ b/apps/server/src/moves/threadMoveView.ts @@ -0,0 +1,69 @@ +/** + * What the move checks (threadMoveRefusals) read about one thread, and no + * more: the thread, the threads that could hold its move (its attached + * children, and threads working in the same checkout), and their projects. + * Loading the whole command read model for this costs about half a second + * on an install with 3,000 threads; this costs a few milliseconds. + */ +import type { + OrchestrationProjectShell, + OrchestrationThread, + ThreadId, +} from "@threadlines/contracts"; +import * as Effect from "effect/Effect"; +import * as Option from "effect/Option"; +import * as SqlClient from "effect/sql/SqlClient"; + +import { ProjectionSnapshotQuery } from "../orchestration/Services/ProjectionSnapshotQuery.ts"; +import type { ThreadMoveView } from "../orchestration/threadMoveDecisions.ts"; + +export interface ThreadMoveContext { + readonly thread: OrchestrationThread; + readonly view: ThreadMoveView; +} + +/** + * Reads one thread's move context; null when the thread is gone. Archived + * threads are read too: a move runs on while its thread is archived. + */ +export const makeThreadMoveContextReader = Effect.gen(function* () { + const sql = yield* SqlClient.SqlClient; + const snapshots = yield* ProjectionSnapshotQuery; + + const projectOf = (projectId: OrchestrationThread["projectId"]) => + snapshots.getProjectShellById(projectId).pipe(Effect.map(Option.getOrUndefined)); + + return (threadId: ThreadId) => + Effect.gen(function* () { + const found = yield* snapshots.getThreadDetailById(threadId, { includeArchived: true }); + if (Option.isNone(found)) return null; + const thread = found.value; + const project = yield* projectOf(thread.projectId); + const checkout = thread.worktreePath ?? project?.workspaceRoot ?? null; + // Its attached children, and every other thread in the same checkout. + const neighbors = yield* sql<{ readonly threadId: ThreadId }>` + SELECT t.thread_id AS "threadId" + FROM projection_threads t + LEFT JOIN projection_projects p ON p.project_id = t.project_id + WHERE t.deleted_at IS NULL + AND t.thread_id <> ${threadId} + AND ( + (t.parent_thread_id = ${threadId} AND t.attached_to_parent = 1) + OR COALESCE(t.worktree_path, p.workspace_root) = ${checkout} + ) + `; + const threads: OrchestrationThread[] = [thread]; + for (const row of neighbors) { + const neighbor = yield* snapshots.getThreadDetailById(row.threadId, { + includeArchived: true, + }); + if (Option.isSome(neighbor)) threads.push(neighbor.value); + } + const projects: OrchestrationProjectShell[] = []; + for (const projectId of new Set(threads.map((entry) => entry.projectId))) { + const shell = projectId === thread.projectId ? project : yield* projectOf(projectId); + if (shell !== undefined) projects.push(shell); + } + return { thread, view: { projects, threads } } satisfies ThreadMoveContext; + }); +}); diff --git a/apps/server/src/orchestration/threadMoveDecisions.ts b/apps/server/src/orchestration/threadMoveDecisions.ts index b33bef264..e76721eeb 100644 --- a/apps/server/src/orchestration/threadMoveDecisions.ts +++ b/apps/server/src/orchestration/threadMoveDecisions.ts @@ -106,7 +106,12 @@ const blocked = (code: ThreadMoveBlockCode, words: string): ThreadMoveRefusalEnt }); /** What the move checks read: the projects, and the threads they can see. */ -export type ThreadMoveView = Pick; +export interface ThreadMoveView { + readonly projects: ReadonlyArray< + Pick + >; + readonly threads: ReadonlyArray; +} /** A thread's checkout: its worktree, else its project's folder. */ function checkoutOf(readModel: ThreadMoveView, thread: OrchestrationThread) { @@ -292,7 +297,7 @@ export function firstMoveBlock(refusals: ReadonlyArray) * it can: a move that doesn't wait (`moves.start`) refuses waits too. */ export function threadMoveRefusal( - readModel: OrchestrationReadModel, + readModel: ThreadMoveView, thread: OrchestrationThread, ): string | null { const refusals = threadMoveRefusals(readModel, thread); From abb0aac3aa3518b827c04d9d00bb0f60a83f21fa Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:50:27 -0500 Subject: [PATCH 22/48] feat(server): the server says it queues moves Capabilities gain threadMoveQueue: a client that sees it may use moves.queue, moves.unqueue and moves.retry and send every move through the queue, since a free thread's move starts at once. Without it, moves.start. --- apps/server/src/environment/Layers/ServerEnvironment.ts | 1 + packages/contracts/src/environment.ts | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/apps/server/src/environment/Layers/ServerEnvironment.ts b/apps/server/src/environment/Layers/ServerEnvironment.ts index a17a408fa..3f2b8834d 100644 --- a/apps/server/src/environment/Layers/ServerEnvironment.ts +++ b/apps/server/src/environment/Layers/ServerEnvironment.ts @@ -102,6 +102,7 @@ export const makeServerEnvironment = Effect.fn("makeServerEnvironment")(function themeShowcaseTakeBack: true, eventTasks: true, threadMove: true, + threadMoveQueue: true, projectSetup: true, stallReports: true, pullRequestObservations: true, diff --git a/packages/contracts/src/environment.ts b/packages/contracts/src/environment.ts index f8c291d2c..5e6322f08 100644 --- a/packages/contracts/src/environment.ts +++ b/packages/contracts/src/environment.ts @@ -81,6 +81,13 @@ export const ExecutionEnvironmentCapabilities = Schema.Struct({ themeShowcaseTakeBack: Schema.optionalKey(Schema.Boolean), /** The server moves threads to linked computers. Absent until a server ships it. */ threadMove: Schema.optionalKey(Schema.Boolean), + /** + * The server queues moves (`moves.queue`, `moves.unqueue`, `moves.retry`): + * a move waits for a working thread to be free, and a free thread's starts + * at once, so a client may send every move through the queue. Absent: only + * `moves.start`. + */ + threadMoveQueue: Schema.optionalKey(Schema.Boolean), /** The server sets up projects copied from another computer (`projectSetups.*`). */ projectSetup: Schema.optionalKey(Schema.Boolean), /** The server reports when it froze (`subscribeServerStalls`). */ From 3f4596dbc544bd8a1608596b20af54c0424bb561 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:50:27 -0500 Subject: [PATCH 23/48] feat(server): a queued move picks its agent when it starts agent is optional in moves.queue, thread.move.queue and pendingMove. Left out, the move takes the agent the target suggests for the thread when it starts (as the dialog picks it), so a move queued while the target is offline still gets one; a target that suggests none ends the move as blocked: " has no agent ready." A named agent is kept. thread_move leaves the agent out unless the caller names one. --- apps/server/src/mcp/McpRoomServer.ts | 7 +- .../server/src/mcp/threadToolHandlers.test.ts | 31 ++++---- apps/server/src/mcp/threadToolHandlers.ts | 25 ++----- apps/server/src/mcp/threadTools.ts | 2 +- apps/server/src/moves/MoveQueue.ts | 3 +- apps/server/src/moves/ThreadMoves.test.ts | 70 +++++++++++++++++++ apps/server/src/moves/ThreadMoves.ts | 60 +++++++++------- .../src/orchestration/threadMoveDecisions.ts | 2 +- packages/contracts/src/moves.ts | 8 ++- packages/contracts/src/orchestration.ts | 10 ++- 10 files changed, 142 insertions(+), 76 deletions(-) create mode 100644 apps/server/src/moves/ThreadMoves.test.ts diff --git a/apps/server/src/mcp/McpRoomServer.ts b/apps/server/src/mcp/McpRoomServer.ts index 7a0de9b5a..faae6e2e9 100644 --- a/apps/server/src/mcp/McpRoomServer.ts +++ b/apps/server/src/mcp/McpRoomServer.ts @@ -46,7 +46,7 @@ import { makeThreadToolLinks } from "../links/threadToolLinks.ts"; import { RepositoryIdentityResolver } from "../project/Services/RepositoryIdentityResolver.ts"; import { ScheduledTasks } from "../scheduled/ScheduledTasks.ts"; import { MoveQueue } from "../moves/MoveQueue.ts"; -import { ThreadMoves } from "../moves/ThreadMoves.ts"; + import { listOnlyGrantedTools, MCP_ROOM_ROUTE_PATH, @@ -246,7 +246,6 @@ const RoomToolHandlersLive = RoomEndpointToolkit.toLayer( // Scheduled tasks, when this server has them (docs/design/fleet.md, Part 6). const scheduledTasks = yield* Effect.serviceOption(ScheduledTasks); // Moving a thread to a linked computer, when this server has links (Part 3b). - const threadMoves = yield* Effect.serviceOption(ThreadMoves); const moveQueue = yield* Effect.serviceOption(MoveQueue); // The user's default for new threads: the last reasoning an agent-made thread follows. const userDefaultModel = settings.getSettings.pipe( @@ -285,9 +284,7 @@ const RoomToolHandlersLive = RoomEndpointToolkit.toLayer( ...(Option.isSome(scheduledTasks) ? { scheduleAnswerWaiting: scheduledTasks.value.waitingThreadIds } : {}), - ...(Option.isSome(threadMoves) && Option.isSome(moveQueue) - ? { moves: { preview: threadMoves.value.preview, queue: moveQueue.value.queue } } - : {}), + ...(Option.isSome(moveQueue) ? { moves: { queue: moveQueue.value.queue } } : {}), }); const scheduleHandlers = makeScheduleToolHandlers({ scheduledTasks: Option.getOrUndefined(scheduledTasks), diff --git a/apps/server/src/mcp/threadToolHandlers.test.ts b/apps/server/src/mcp/threadToolHandlers.test.ts index f7ff2f2de..185bd63e6 100644 --- a/apps/server/src/mcp/threadToolHandlers.test.ts +++ b/apps/server/src/mcp/threadToolHandlers.test.ts @@ -1386,24 +1386,10 @@ describe("thread tools across linked computers", () => { }); describe("thread_move", () => { - /** Moves that record what was queued, to a Mac mini that suggests its Claude. */ + /** Moves that record what was queued. */ const fakeMoves = () => { const queued: Array[0]> = []; const moves: ThreadToolMoves = { - preview: () => - Effect.succeed({ - targets: [ - { - environmentId: MAC_MINI, - label: "Mac mini", - agents: MAC_DESCRIPTOR.agents, - suggestedAgent: { - instanceId: ProviderInstanceId.make("mac-claude"), - model: "opus-5-5", - }, - }, - ], - }), queue: (input) => Effect.sync(() => { queued.push(input); @@ -1427,15 +1413,28 @@ describe("thread_move", () => { detail: "Queued: this thread moves to Mac mini when your turn ends. Finish your reply; don't start more work.", }); + // No agent named: the one Mac mini suggests when the move starts. expect(queued).toEqual([ { threadId: PARENT, targetEnvironmentId: MAC_MINI, - agent: { instanceId: "mac-claude", model: "opus-5-5" }, mode: "when-done", requestedBy: { kind: "agent", turnId: TURN }, }, ]); + + // A named agent goes with it. + const named = fakeMoves(); + const again = yield* makeTools({ + links: fakeLinks(), + moves: named.moves, + caller: makeThread({ deletedAt: null }), + }); + yield* again.handlers.thread_move(mainCaller(), { + computer: "Mac mini", + agent: { instanceId: "mac-codex", model: "gpt-6-astra" }, + }); + expect(named.queued[0]?.agent).toEqual({ instanceId: "mac-codex", model: "gpt-6-astra" }); }), ); diff --git a/apps/server/src/mcp/threadToolHandlers.ts b/apps/server/src/mcp/threadToolHandlers.ts index 17269198b..463b5bc83 100644 --- a/apps/server/src/mcp/threadToolHandlers.ts +++ b/apps/server/src/mcp/threadToolHandlers.ts @@ -45,7 +45,6 @@ import { type ServerProvider, ThreadId, type ThreadMoveError, - type ThreadMovePreview, type ThreadMoveQueueInput, type ThreadMoveRequester, TurnId, @@ -152,9 +151,8 @@ export interface ThreadToolDeps { readonly moves?: ThreadToolMoves; } -/** What `thread_move` needs of the moves (ThreadMoves.preview, MoveQueue.queue). */ +/** What `thread_move` needs of the moves (MoveQueue.queue, which checks the target). */ export interface ThreadToolMoves { - readonly preview: (threadId: ThreadId) => Effect.Effect; readonly queue: ( input: ThreadMoveQueueInput & { readonly requestedBy: ThreadMoveRequester }, ) => Effect.Effect; @@ -2092,34 +2090,19 @@ export function makeThreadToolHandlers(deps: ThreadToolDeps) { ) { return { outcome: "queued", detail: queuedDetail } satisfies ThreadMoveResult; } - const preview = yield* deps.moves.preview(thread.id).pipe(Effect.result); - if (preview._tag === "Failure") { - return { outcome: "refused", detail: preview.failure.detail } satisfies ThreadMoveResult; - } - if (preview.success.refusal !== undefined) { - return { outcome: "refused", detail: preview.success.refusal } satisfies ThreadMoveResult; - } - const target = preview.success.targets.find( - (entry) => entry.environmentId === computer.environmentId, - ); + // No agent named: the one that computer suggests when the move starts. const agent: ModelSelection | undefined = input.agent !== undefined ? ({ instanceId: ProviderInstanceId.make(input.agent.instanceId.trim()), model: input.agent.model.trim(), } as ModelSelection) - : target?.suggestedAgent; - if (agent === undefined) { - return { - outcome: "refused", - detail: `${computer.label} can't be reached right now, so there's no agent to suggest there. Name one of its agents from thread_agents' \`computers\`, or try again once it connects.`, - } satisfies ThreadMoveResult; - } + : undefined; const queued = yield* deps.moves .queue({ threadId: thread.id, targetEnvironmentId: computer.environmentId, - agent, + ...(agent !== undefined ? { agent } : {}), mode: "when-done", requestedBy: { kind: "agent", turnId: latched.turnId }, }) diff --git a/apps/server/src/mcp/threadTools.ts b/apps/server/src/mcp/threadTools.ts index 4afbf318d..18df77ed2 100644 --- a/apps/server/src/mcp/threadTools.ts +++ b/apps/server/src/mcp/threadTools.ts @@ -332,7 +332,7 @@ export const ThreadStopTool = drivesThreads( export const ThreadMoveTool = drivesThreads( Tool.make("thread_move", { description: - "Move this thread to another linked computer, to carry on there (the user may be shutting this one down). It moves when your turn ends, with its messages and code changes; you continue there in a fresh session with the conversation so far. Only your own thread. `computer` is a name or id from thread_agents' `computers`; `agent` is one of that computer's agents, left out for the one it suggests. The user can cancel it, and that computer may ask its user first. After calling it, finish your reply and don't start more work.", + "Move this thread to another linked computer, to carry on there (the user may be shutting this one down). It moves when your turn ends, with its messages and code changes; you continue there in a fresh session with the conversation so far. Only your own thread. `computer` is a name or id from thread_agents' `computers`; `agent` is one of that computer's agents, left out for the one it suggests when the move starts. The user can cancel it, and that computer may ask its user first. After calling it, finish your reply and don't start more work.", parameters: Schema.Struct({ computer: Schema.String.annotate({ description: "The linked computer: a name or id from thread_agents' `computers`.", diff --git a/apps/server/src/moves/MoveQueue.ts b/apps/server/src/moves/MoveQueue.ts index 156686a65..bc6c94c0a 100644 --- a/apps/server/src/moves/MoveQueue.ts +++ b/apps/server/src/moves/MoveQueue.ts @@ -383,6 +383,7 @@ export const makeMoveQueue = ( // The stop is still landing in the projection; its event looks again. return yield* schedule(threadId, retryDelay(0)); } + // No agent named: the target's suggestion now, or it can't go. const outcome: QueuedMoveStart = yield* deps.startQueued({ threadId, requestId, @@ -426,7 +427,7 @@ export const makeMoveQueue = ( threadId: input.threadId, requestId, target: { environmentId: input.targetEnvironmentId, label }, - agent: input.agent, + ...(input.agent !== undefined ? { agent: input.agent } : {}), mode: input.mode, requestedBy: input.requestedBy ?? { kind: "user" }, createdAt: yield* nowIso, diff --git a/apps/server/src/moves/ThreadMoves.test.ts b/apps/server/src/moves/ThreadMoves.test.ts new file mode 100644 index 000000000..3906e78fc --- /dev/null +++ b/apps/server/src/moves/ThreadMoves.test.ts @@ -0,0 +1,70 @@ +import { + EnvironmentId, + type ModelSelection, + ProviderInstanceId, + type ThreadMovePreview, +} from "@threadlines/contracts"; +import { describe, expect, it } from "vite-plus/test"; + +import { checkMoveTarget } from "./ThreadMoves.ts"; + +const MAC_MINI = EnvironmentId.make("environment-mac-mini"); +const claude: ModelSelection = { + instanceId: ProviderInstanceId.make("mac-claude"), + model: "opus-5-5", +}; +const codex: ModelSelection = { + instanceId: ProviderInstanceId.make("mac-codex"), + model: "gpt-6-astra", +}; +const agents = [ + { + instanceId: "mac-claude", + name: "Claude", + billing: "Claude Max", + models: [{ model: "opus-5-5", name: "Opus 5.5" }], + driver: "claudeAgent", + }, + { + instanceId: "mac-codex", + name: "Codex", + billing: "ChatGPT Pro", + models: [{ model: "gpt-6-astra", name: "GPT-6 Astra" }], + driver: "codex", + }, +] as ThreadMovePreview["targets"][number]["agents"]; + +const shown = (target: Partial): ThreadMovePreview => ({ + targets: [{ environmentId: MAC_MINI, label: "Mac mini", agents, ...target }], +}); + +describe("checkMoveTarget", () => { + it("takes the target's suggestion when the move names no agent, and a named one as named", () => { + expect(checkMoveTarget(shown({ suggestedAgent: claude }), MAC_MINI, undefined)).toMatchObject({ + agent: claude, + }); + expect(checkMoveTarget(shown({ suggestedAgent: claude }), MAC_MINI, codex)).toMatchObject({ + agent: codex, + }); + }); + + it("can't go when the target suggests none and none is named", () => { + expect(checkMoveTarget(shown({}), MAC_MINI, undefined)).toEqual({ + refusal: "Mac mini has no agent ready.", + }); + }); + + it("waits on an unreachable target, whatever the agent", () => { + expect( + checkMoveTarget( + shown({ + refusal: "Mac mini can't be reached right now.", + refusalCode: "unreachable", + agents: [], + }), + MAC_MINI, + undefined, + ), + ).toHaveProperty("unreachable"); + }); +}); diff --git a/apps/server/src/moves/ThreadMoves.ts b/apps/server/src/moves/ThreadMoves.ts index 18a824d86..c1c7b2818 100644 --- a/apps/server/src/moves/ThreadMoves.ts +++ b/apps/server/src/moves/ThreadMoves.ts @@ -103,14 +103,16 @@ export interface ThreadMovesShape { readonly checkQueue: (input: { readonly threadId: ThreadId; readonly targetEnvironmentId: EnvironmentId; - readonly agent: ModelSelection; + /** Absent: the target's suggested agent, checked when the move starts. */ + readonly agent?: ModelSelection | undefined; }) => Effect.Effect<{ readonly label: string }, ThreadMoveError>; /** Starts the thread's queued move `requestId`, now that the thread is free and quiesced. */ readonly startQueued: (input: { readonly threadId: ThreadId; readonly requestId: ThreadMoveRequestId; readonly targetEnvironmentId: EnvironmentId; - readonly agent: ModelSelection; + /** Absent: the agent the target suggests now. */ + readonly agent?: ModelSelection | undefined; }) => Effect.Effect; readonly cancel: (threadId: ThreadId) => Effect.Effect; /** Run unfinished moves to their end, and every move started from now on. */ @@ -148,6 +150,32 @@ function suggestAgent( : undefined; } +/** + * The target and agent a move names, checked against what the target said + * (`moves.preview`). No agent named: the one the target suggests for this + * thread, as the dialog picks it; none there means the move can't go. + */ +export function checkMoveTarget( + shown: ThreadMovePreview, + environmentId: EnvironmentId, + named: ModelSelection | undefined, +): + | { readonly ok: ThreadMoveTargetPreview; readonly agent: ModelSelection } + | { readonly unreachable: ThreadMoveTargetPreview } + | { readonly refusal: string } { + const target = shown.targets.find((entry) => entry.environmentId === environmentId); + if (target === undefined) return { refusal: "This computer isn't linked to that one." }; + if (target.refusalCode === "unreachable") return { unreachable: target }; + if (target.refusal !== undefined) return { refusal: target.refusal }; + const agent = named ?? target.suggestedAgent; + if (agent === undefined) return { refusal: `${target.label} has no agent ready.` }; + const offered = target.agents.find((entry) => entry.instanceId === agent.instanceId); + if (offered === undefined || !offered.models.some((model) => model.model === agent.model)) { + return { refusal: `${target.label} has no such agent or model.` }; + } + return { ok: target, agent }; +} + const make = Effect.gen(function* () { const engine = yield* OrchestrationEngineService; const snapshots = yield* ProjectionSnapshotQuery; @@ -309,26 +337,6 @@ const make = Effect.gen(function* () { const preview: ThreadMovesShape["preview"] = (threadId) => inspect(threadId).pipe(Effect.map((inspected) => inspected.preview)); - /** The target and agent a move names, checked against what the target said. */ - const checkTarget = ( - shown: ThreadMovePreview, - environmentId: EnvironmentId, - agent: ModelSelection, - ): - | { readonly ok: ThreadMoveTargetPreview } - | { readonly unreachable: ThreadMoveTargetPreview } - | { readonly refusal: string } => { - const target = shown.targets.find((entry) => entry.environmentId === environmentId); - if (target === undefined) return { refusal: "This computer isn't linked to that one." }; - if (target.refusalCode === "unreachable") return { unreachable: target }; - if (target.refusal !== undefined) return { refusal: target.refusal }; - const offered = target.agents.find((entry) => entry.instanceId === agent.instanceId); - if (offered === undefined || !offered.models.some((model) => model.model === agent.model)) { - return { refusal: `${target.label} has no such agent or model.` }; - } - return { ok: target }; - }; - /** Freezes the thread for its move (`thread.move.start`); the decider checks it again. */ const dispatchStart = ( threadId: ThreadId, @@ -357,7 +365,7 @@ const make = Effect.gen(function* () { const shown = inspected.preview; if (shown.refusal !== undefined) return yield* moveError(shown.refusal); if (inspected.busy !== null) return yield* moveError(inspected.busy); - const checked = checkTarget(shown, input.targetEnvironmentId, input.agent); + const checked = checkMoveTarget(shown, input.targetEnvironmentId, input.agent); if ("refusal" in checked) return yield* moveError(checked.refusal); if ("unreachable" in checked) { return yield* moveError(checked.unreachable.refusal ?? "That computer can't be reached."); @@ -372,7 +380,7 @@ const make = Effect.gen(function* () { Effect.gen(function* () { const shown = (yield* inspect(input.threadId)).preview; if (shown.refusal !== undefined) return yield* moveError(shown.refusal); - const checked = checkTarget(shown, input.targetEnvironmentId, input.agent); + const checked = checkMoveTarget(shown, input.targetEnvironmentId, input.agent); if ("refusal" in checked) return yield* moveError(checked.refusal); return { label: "ok" in checked ? checked.ok.label : checked.unreachable.label }; }); @@ -385,7 +393,7 @@ const make = Effect.gen(function* () { } const shown = inspected.success.preview; if (shown.refusal !== undefined) return { kind: "blocked", words: shown.refusal } as const; - const checked = checkTarget(shown, input.targetEnvironmentId, input.agent); + const checked = checkMoveTarget(shown, input.targetEnvironmentId, input.agent); if ("refusal" in checked) return { kind: "blocked", words: checked.refusal } as const; if ("unreachable" in checked) { return { @@ -396,7 +404,7 @@ const make = Effect.gen(function* () { const started = yield* dispatchStart( input.threadId, checked.ok, - input.agent, + checked.agent, input.requestId, ).pipe(Effect.result); return started._tag === "Success" diff --git a/apps/server/src/orchestration/threadMoveDecisions.ts b/apps/server/src/orchestration/threadMoveDecisions.ts index e76721eeb..0f6b236cc 100644 --- a/apps/server/src/orchestration/threadMoveDecisions.ts +++ b/apps/server/src/orchestration/threadMoveDecisions.ts @@ -593,7 +593,7 @@ export function decideMoveQueue( pendingMove: { requestId: command.requestId, target: command.target, - agent: command.agent, + ...(command.agent !== undefined ? { agent: command.agent } : {}), mode: command.mode, stage: command.mode === "stop-now" ? "quiescing" : "waiting", requestedAt: command.createdAt, diff --git a/packages/contracts/src/moves.ts b/packages/contracts/src/moves.ts index 8f79ece68..b82a5180e 100644 --- a/packages/contracts/src/moves.ts +++ b/packages/contracts/src/moves.ts @@ -94,8 +94,12 @@ export type ThreadMoveCancelInput = typeof ThreadMoveCancelInput.Type; export const ThreadMoveQueueInput = Schema.Struct({ threadId: ThreadId, targetEnvironmentId: EnvironmentId, - /** One of the target's agents, by its own instance id. */ - agent: ModelSelection, + /** + * One of the target's agents, by its own instance id. Absent: the agent + * the target suggests when the move starts (as the dialog picks it), so a + * move to a computer that is offline now still gets one. + */ + agent: Schema.optional(ModelSelection), mode: ThreadPendingMoveMode, }); export type ThreadMoveQueueInput = typeof ThreadMoveQueueInput.Type; diff --git a/packages/contracts/src/orchestration.ts b/packages/contracts/src/orchestration.ts index 614ade03f..a0be910d2 100644 --- a/packages/contracts/src/orchestration.ts +++ b/packages/contracts/src/orchestration.ts @@ -883,8 +883,11 @@ export type ThreadMoveHold = typeof ThreadMoveHold.Type; export const ThreadPendingMove = Schema.Struct({ requestId: ThreadMoveRequestId, target: ThreadMoveTarget, - /** The agent the other computer will run it with (its own instance id). */ - agent: ModelSelection, + /** + * The agent the other computer will run it with (its own instance id). + * Absent: the one that computer suggests when the move starts. + */ + agent: Schema.optional(ModelSelection), mode: ThreadPendingMoveMode, stage: ThreadPendingMoveStage, requestedAt: IsoDateTime, @@ -4808,7 +4811,8 @@ const ThreadMoveQueueCommand = Schema.Struct({ threadId: ThreadId, requestId: ThreadMoveRequestId, target: ThreadMoveTarget, - agent: ModelSelection, + /** Absent: the target's suggested agent, picked when the move starts. */ + agent: Schema.optional(ModelSelection), mode: ThreadPendingMoveMode, requestedBy: ThreadMoveRequester, createdAt: IsoDateTime, From 4318edfeeb458bbe7a829839e0c9d1df110b81aa Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 23:30:28 -0500 Subject: [PATCH 24/48] fix(server): a start or recovery under way can't open a session after a move freezes quiesceThread counted only listed sessions and early starts, so a recovery that had passed the session fence and was still opening its runtime went unseen: the move confirmed, froze, and then the runtime came up. It now waits for every start or recovery of the thread that holds its key's gate, then stops what they opened. Recovery reads the fence holding the gate, as starts already did, so one that waited behind the move is refused. --- .../provider/Layers/ProviderService.test.ts | 42 ++++++++++ .../src/provider/Layers/ProviderService.ts | 78 ++++++++++++------- 2 files changed, 93 insertions(+), 27 deletions(-) diff --git a/apps/server/src/provider/Layers/ProviderService.test.ts b/apps/server/src/provider/Layers/ProviderService.test.ts index ac01ac1fd..cb3abe238 100644 --- a/apps/server/src/provider/Layers/ProviderService.test.ts +++ b/apps/server/src/provider/Layers/ProviderService.test.ts @@ -3193,4 +3193,46 @@ fence.layer("ProviderServiceLive moving threads", (it) => { }); }), ); + + it.effect("waits for a recovery already under way, and stops what it opened", () => + Effect.gen(function* () { + const provider = yield* ProviderService; + const RECOVERING = asThreadId("8c2e4f6a-1b3d-4e5f-9a7b-2c4d6e8f0a1b"); + yield* startOn(codexInstanceId, RECOVERING); + // The runtime is gone; a transcript read will bring it back. + yield* fence.codex.stopSession(RECOVERING); + const reached = yield* Deferred.make(); + const release = yield* Deferred.make(); + const startNormally = fence.codex.startSession.getMockImplementation()!; + fence.codex.startSession.mockImplementationOnce((input) => + Deferred.succeed(reached, undefined).pipe( + Effect.andThen(Deferred.await(release)), + Effect.andThen(startNormally(input)), + ), + ); + const recovery = yield* provider + .rollbackConversation({ threadId: RECOVERING, numTurns: 1 }) + .pipe(Effect.exit, Effect.forkChild); + // It passed the fence and is opening the runtime when the move begins. + yield* Deferred.await(reached); + fence.fenced.add(RECOVERING); + const quiesce = yield* provider.quiesceThread(RECOVERING).pipe(Effect.forkChild); + for (let turn = 0; turn < 20; turn += 1) yield* Effect.yieldNow; + assert.isUndefined(quiesce.pollUnsafe()); + + yield* Deferred.succeed(release, undefined); + yield* Fiber.join(recovery); + assert.deepEqual(yield* Fiber.join(quiesce), { confirmed: true }); + const left = (yield* provider.listSessions()).filter( + (session) => session.threadId === RECOVERING, + ); + assert.deepEqual(left, []); + // Any recovery from here is refused. + const later = yield* provider + .rollbackConversation({ threadId: RECOVERING, numTurns: 1 }) + .pipe(Effect.result); + assert.equal(later._tag, "Failure"); + fence.fenced.delete(RECOVERING); + }), + ); }); diff --git a/apps/server/src/provider/Layers/ProviderService.ts b/apps/server/src/provider/Layers/ProviderService.ts index c703e8228..aea07ba07 100644 --- a/apps/server/src/provider/Layers/ProviderService.ts +++ b/apps/server/src/provider/Layers/ProviderService.ts @@ -84,6 +84,8 @@ import { type EarlySessionEndReason, makeEarlySessionLedger } from "../earlySess const EARLY_SESSION_START_TIMEOUT = Duration.seconds(60); /** How often unused early sessions are checked for expiry. */ const EARLY_SESSION_SWEEP_INTERVAL = Duration.seconds(30); +/** How long quiesceThread waits for a start or recovery of the thread already under way. */ +const QUIESCE_START_WAIT = Duration.seconds(90); import { AnalyticsService } from "../../telemetry/Services/AnalyticsService.ts"; import { analyticsAgentProperties, @@ -745,7 +747,6 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( readonly binding: ProviderRuntimeBinding; readonly operation: string; }) { - yield* requireAdmission(input.operation, input.binding.threadId); const bindingInstanceId = yield* requireBindingInstanceId(input.operation, input.binding); yield* Effect.annotateCurrentSpan({ "provider.operation": "recover-session", @@ -758,6 +759,9 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( return yield* withEarlyGate( input.binding.threadId, Effect.gen(function* () { + // The fence is read holding the gate, so a recovery that waited on + // it while a move began is refused (quiesceThread waits for gates). + yield* requireAdmission(input.operation, input.binding.threadId); if (earlySessions.waiting(input.binding.threadId) === undefined) { earlySessions.forget(input.binding.threadId); } @@ -3022,33 +3026,53 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( ), ), ).pipe(Effect.map((perAdapter) => perAdapter.flat())); - const live = yield* liveOfThread; - const ran = new Set(live.map((entry) => entry.instanceId)); - // Early starts still starting have no session to list yet. One whose - // agent already runs has what it started noted before it is ended; a - // listed session's stop notes its own (stopSessionCore). - const keys = new Set(live.map((entry) => entry.key)); - for (const rawKey of earlyStartRuns.keys()) { - if (!ofThread(rawKey)) continue; - const key = ThreadId.make(rawKey); - keys.add(key); - if (live.some((entry) => entry.key === key)) continue; - for (const [, adapter] of yield* getAdapterEntries) { - if (adapter.noteBeforeStop !== undefined) yield* adapter.noteBeforeStop(key); - } - } + const ran = new Set(); const problems: string[] = []; - for (const key of keys) { - yield* stopSession({ threadId: key }).pipe( - Effect.catchCause((cause) => - Effect.sync(() => { - const error = Cause.squash(cause); - problems.push( - `stopping it failed (${error instanceof Error ? error.message : String(error)})`, - ); - }), - ), - ); + /** Stops every listed session of the thread, and its early starts. */ + const stopWhatRuns = Effect.gen(function* () { + const live = yield* liveOfThread; + for (const entry of live) ran.add(entry.instanceId); + // Early starts still starting have no session to list yet. One whose + // agent already runs has what it started noted before it is ended; a + // listed session's stop notes its own (stopSessionCore). + const keys = new Set(live.map((entry) => entry.key)); + for (const rawKey of earlyStartRuns.keys()) { + if (!ofThread(rawKey)) continue; + const key = ThreadId.make(rawKey); + keys.add(key); + if (live.some((entry) => entry.key === key)) continue; + for (const [, adapter] of yield* getAdapterEntries) { + if (adapter.noteBeforeStop !== undefined) yield* adapter.noteBeforeStop(key); + } + } + for (const key of keys) { + yield* stopSession({ threadId: key }).pipe( + Effect.catchCause((cause) => + Effect.sync(() => { + const error = Cause.squash(cause); + problems.push( + `stopping it failed (${error instanceof Error ? error.message : String(error)})`, + ); + }), + ), + ); + } + }); + yield* stopWhatRuns; + // A start or recovery that passed the fence before it went up holds + // its key's gate until its session is listed: wait for each, then stop + // what they opened. Any start from here is refused (it reads the fence + // holding the gate). + const inFlight = [...earlyGates.entries()] + .filter(([key]) => ofThread(key)) + .map(([, gate]) => gate); + if (inFlight.length > 0) { + const settled = yield* Effect.forEach(inFlight, (gate) => Deferred.await(gate), { + concurrency: "unbounded", + discard: true, + }).pipe(Effect.timeoutOption(QUIESCE_START_WAIT)); + if (Option.isNone(settled)) problems.push("a session of it is still starting"); + yield* stopWhatRuns; } for (const entry of yield* liveOfThread) { problems.push(`${entry.adapter.provider} still lists its session`); From 614c9a137f6452753beaddf28d38d2e61517e002 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 23:34:08 -0500 Subject: [PATCH 25/48] fix(opencode): a restart no longer passes for proof that OpenCode stopped What a stop interrupted lived only in memory, so after a restart an empty record confirmed a thread whose work the shared server may still run (an external server outlives us). confirmThreadStopped now gets what the session directory keeps of each of the thread's sessions; for one this process never ran, OpenCode asks the server whether its root session or any child is active and whether a shell runs in its folder, and confirms only when nothing does. --- .../src/provider/Layers/ClaudeAdapter.test.ts | 2 +- .../provider/Layers/OpenCodeAdapter.test.ts | 48 +++++++++++++- .../src/provider/Layers/OpenCodeAdapter.ts | 64 +++++++++++++++++++ .../provider/Layers/ProviderService.test.ts | 16 ++++- .../src/provider/Layers/ProviderService.ts | 22 ++++++- .../src/provider/Services/ProviderAdapter.ts | 17 ++++- 6 files changed, 159 insertions(+), 10 deletions(-) diff --git a/apps/server/src/provider/Layers/ClaudeAdapter.test.ts b/apps/server/src/provider/Layers/ClaudeAdapter.test.ts index b60b56306..91357e18d 100644 --- a/apps/server/src/provider/Layers/ClaudeAdapter.test.ts +++ b/apps/server/src/provider/Layers/ClaudeAdapter.test.ts @@ -6153,7 +6153,7 @@ describe("ClaudeAdapterLive", () => { }); yield* adapter.noteBeforeStop!(THREAD_ID); yield* adapter.stopSession(THREAD_ID); - return yield* adapter.confirmThreadStopped!(THREAD_ID); + return yield* adapter.confirmThreadStopped!(THREAD_ID, []); }).pipe(Effect.provideService(Random.Random, makeDeterministicRandomService())); // The SDK says which process it started: the stop is proven, and a diff --git a/apps/server/src/provider/Layers/OpenCodeAdapter.test.ts b/apps/server/src/provider/Layers/OpenCodeAdapter.test.ts index 3f351eb1e..c18899b17 100644 --- a/apps/server/src/provider/Layers/OpenCodeAdapter.test.ts +++ b/apps/server/src/provider/Layers/OpenCodeAdapter.test.ts @@ -70,7 +70,11 @@ function fakeClient(calls: Array, overrides: Record ({ data: [] })) }, permission: { reply: call("permission.reply"), list: call("permission.list", () => []) }, - shell: { remove: call("shell.remove"), get: call("shell.get") }, + shell: { + remove: call("shell.remove"), + get: call("shell.get"), + list: call("shell.list", () => ({ location: {}, data: [] })), + }, message: { list: call("message.list", () => ({ data: [], cursor: {} })) }, session: { create: call("session.create", () => ({ id: ROOT, location: { directory: "/" } })), @@ -349,15 +353,53 @@ describe("OpenCodeAdapter", () => { yield* next((event) => event.type === "task.started"); yield* adapter.stopSession(threadId); - expect(yield* adapter.confirmThreadStopped!(threadId)).toEqual({ confirmed: true }); + expect(yield* adapter.confirmThreadStopped!(threadId, [])).toEqual({ confirmed: true }); expect(shellReads).toBe(2); expect(calls.some((call) => call.method === "session.active")).toBe(true); // Proven once, nothing is left to ask about. - expect(yield* adapter.confirmThreadStopped!(threadId)).toEqual({ confirmed: true }); + expect(yield* adapter.confirmThreadStopped!(threadId, [])).toEqual({ confirmed: true }); expect(shellReads).toBe(2); }).pipe(Effect.scoped, Effect.provide(testLayer)), ); + it.live("after a restart, asks the server what still runs before it confirms", () => + Effect.gen(function* () { + // This process never ran the thread: what its last stop knew is gone. + let active: Record = { [ROOT]: { type: "running" } }; + let shells: Array<{ status: string }> = [{ status: "running" }]; + const { adapter, calls, threadId, cwd } = yield* harnessWith({ + "session.active": () => active, + "session.list": (input: { parentID: string }) => ({ + data: input.parentID === ROOT ? [{ id: "ses_child" }] : [], + }), + "shell.list": () => ({ location: {}, data: shells }), + }); + const known = [ + { sessionKey: threadId, resumeCursor: { schemaVersion: 1, sessionId: ROOT }, cwd }, + ]; + expect(yield* adapter.confirmThreadStopped!(threadId, known)).toEqual({ + confirmed: false, + reason: "OpenCode still runs its work; OpenCode still runs 1 shell in its folder", + }); + // Its child counts too. + active = { ses_child: { type: "running" } }; + shells = []; + expect(yield* adapter.confirmThreadStopped!(threadId, known)).toEqual({ + confirmed: false, + reason: "OpenCode still runs its work", + }); + active = {}; + expect(yield* adapter.confirmThreadStopped!(threadId, known)).toEqual({ confirmed: true }); + expect( + calls.some( + (call) => + call.method === "shell.list" && + (call.input as { location: { directory: string } }).location.directory === cwd, + ), + ).toBe(true); + }).pipe(Effect.scoped, Effect.provide(testLayer)), + ); + it.live("stops one background shell, and its end is reported once", () => Effect.gen(function* () { const { adapter, calls, send, next, threadId, cwd } = yield* harness; diff --git a/apps/server/src/provider/Layers/OpenCodeAdapter.ts b/apps/server/src/provider/Layers/OpenCodeAdapter.ts index 912df02ab..de04671b4 100644 --- a/apps/server/src/provider/Layers/OpenCodeAdapter.ts +++ b/apps/server/src/provider/Layers/OpenCodeAdapter.ts @@ -433,6 +433,12 @@ export function makeOpenCodeAdapter(options: OpenCodeAdapterOptions) { readonly shellIds: ReadonlyArray; } >(); + /** + * Session keys this process ran. One it never ran may still have work on + * the server from before a restart, whose stop evidence died with the + * process: confirmThreadStopped asks the server about it instead. + */ + const ranHere = new Set(); /** Every OpenCode session we own, root or child, to the thread it belongs to. */ const owners = new Map(); /** Which session registered each Threadlines tool server, so a stale stop never removes a newer one. */ @@ -2415,6 +2421,7 @@ export function makeOpenCodeAdapter(options: OpenCodeAdapterOptions) { } sessions.set(input.threadId, ctx); + ranHere.add(input.threadId); owners.set(rootSessionId, ctx); yield* emit({ @@ -3217,14 +3224,71 @@ export function makeOpenCodeAdapter(options: OpenCodeAdapterOptions) { * waiting a little for interrupts still settling. A thread that was idle * when it stopped left nothing to ask about. */ + /** + * What OpenCode may still run for a session this process never ran (a + * restart lost what its stop knew): its root session and their children, + * and any shell running in its folder. Problems, in words. + */ + const workLeftOnServer = (rootSessionId: string, directory: string | undefined) => + manager + .withServer(({ server }) => + Effect.gen(function* () { + const client = server.client; + const call = (method: string, run: (signal: AbortSignal) => Promise) => + runOpenCode(method, run).pipe(Effect.timeout(OPENCODE_CONFIRM_CALL_TIMEOUT)); + const sessionIds = [rootSessionId]; + for (let index = 0; index < sessionIds.length && sessionIds.length < 200; index++) { + const parentID = sessionIds[index]!; + const children = yield* call("session.list", (signal) => + client.session.list({ parentID, limit: 100 }, { signal }), + ); + sessionIds.push(...(children.data ?? []).map((child) => child.id)); + } + const active = yield* call("session.active", (signal) => + client.session.active({ signal }), + ); + const problems: string[] = []; + if (sessionIds.some((id) => Object.hasOwn(active, id))) { + problems.push("OpenCode still runs its work"); + } + if (directory === undefined) { + problems.push("its folder isn't known, so OpenCode's shells there can't be checked"); + } else { + const shells = yield* call("shell.list", (signal) => + client.shell.list({ location: { directory } }, { signal }), + ); + const running = shells.data.filter((shell) => shell.status === "running").length; + if (running > 0) { + problems.push( + `OpenCode still runs ${running} shell${running === 1 ? "" : "s"} in its folder`, + ); + } + } + return problems; + }), + ) + .pipe( + Effect.catch((error) => + Effect.succeed([`OpenCode's server didn't answer (${openCodeErrorDetail(error)})`]), + ), + ); + const confirmThreadStopped: NonNullable = ( threadId, + known, ) => Effect.gen(function* () { const keys = [...stoppedWork.keys()].filter( (key) => parseSessionKey(key).threadId === threadId, ); const problems: string[] = []; + // Never confirmed on evidence a restart emptied: ask the server. + for (const hint of known) { + if (ranHere.has(hint.sessionKey) || stoppedWork.has(hint.sessionKey)) continue; + const rootSessionId = parseOpenCodeResumeCursor(hint.resumeCursor)?.sessionId; + if (rootSessionId === undefined) continue; + problems.push(...(yield* workLeftOnServer(rootSessionId, hint.cwd))); + } for (const key of keys) { const work = stoppedWork.get(key); if (work === undefined) continue; diff --git a/apps/server/src/provider/Layers/ProviderService.test.ts b/apps/server/src/provider/Layers/ProviderService.test.ts index cb3abe238..01c916de7 100644 --- a/apps/server/src/provider/Layers/ProviderService.test.ts +++ b/apps/server/src/provider/Layers/ProviderService.test.ts @@ -56,6 +56,7 @@ import type { ProviderAdapterShape, ProviderSetThreadGoalInput, ProviderStopConfirmation, + ProviderStoppedSessionHint, } from "../Services/ProviderAdapter.ts"; import { ProviderAdapterRegistry, @@ -3067,8 +3068,10 @@ function makeFencedProviderServiceLayer() { const fenced = new Set(); const noteBeforeStop = vi.fn((_sessionKey: ThreadId): Effect.Effect => Effect.void); const confirmThreadStopped = vi.fn( - (_threadId: ThreadId): Effect.Effect => - Effect.succeed({ confirmed: true }), + ( + _threadId: ThreadId, + _known: ReadonlyArray, + ): Effect.Effect => Effect.succeed({ confirmed: true }), ); const registry = makeAdapterRegistryMock({ [ProviderDriverKind.make("codex")]: { ...codex.adapter, confirmThreadStopped, noteBeforeStop }, @@ -3165,7 +3168,14 @@ fence.layer("ProviderServiceLive moving threads", (it) => { assert.deepEqual(yield* provider.quiesceThread(MOVING), { confirmed: true }); const left = (yield* provider.listSessions()).map((session) => session.threadId); assert.deepEqual(left, [other]); - assert.deepEqual(fence.confirmThreadStopped.mock.calls.at(-1), [MOVING]); + const [confirmedFor, known] = fence.confirmThreadStopped.mock.calls.at(-1)!; + assert.equal(confirmedFor, MOVING); + // With what the directory keeps of each session the driver ran for it, + // for a driver whose own evidence a restart lost. + assert.deepEqual( + known.map((hint) => hint.sessionKey).toSorted(), + [MOVING, participantSessionKey(MOVING, ROOM_AGENT)].toSorted(), + ); // What each stopping runtime started was noted first; a stop that isn't // for a move never reads the process table. assert.deepEqual( diff --git a/apps/server/src/provider/Layers/ProviderService.ts b/apps/server/src/provider/Layers/ProviderService.ts index aea07ba07..a0d958eb5 100644 --- a/apps/server/src/provider/Layers/ProviderService.ts +++ b/apps/server/src/provider/Layers/ProviderService.ts @@ -3077,13 +3077,31 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( for (const entry of yield* liveOfThread) { problems.push(`${entry.adapter.provider} still lists its session`); } - // Each driver proves its processes, and what they started, are gone. + // Each driver proves its processes, and what they started, are gone, + // with what the directory keeps of the sessions it ran for the thread. + const bindings = (yield* directory.listBindings().pipe( + Effect.catch(() => + Effect.sync(() => { + problems.push("its sessions' records couldn't be read"); + return []; + }), + ), + )).filter((binding) => ofThread(binding.threadId)); for (const [instanceId, adapter] of yield* getAdapterEntries) { if (adapter.confirmThreadStopped === undefined) { if (ran.has(instanceId)) problems.push(`${adapter.provider} can't confirm it stopped`); continue; } - const confirmation = yield* adapter.confirmThreadStopped(threadId); + const confirmation = yield* adapter.confirmThreadStopped( + threadId, + bindings + .filter((binding) => binding.providerInstanceId === instanceId) + .map((binding) => ({ + sessionKey: binding.threadId, + resumeCursor: binding.resumeCursor, + cwd: readPersistedCwd(binding.runtimePayload), + })), + ); if (!confirmation.confirmed) problems.push(confirmation.reason); } return problems.length === 0 diff --git a/apps/server/src/provider/Services/ProviderAdapter.ts b/apps/server/src/provider/Services/ProviderAdapter.ts index 7b7c6e3af..94ee0cd16 100644 --- a/apps/server/src/provider/Services/ProviderAdapter.ts +++ b/apps/server/src/provider/Services/ProviderAdapter.ts @@ -467,7 +467,15 @@ export interface ProviderAdapterShape { * Absent: the driver can't confirm, which counts as unconfirmed for a * thread it ran. */ - readonly confirmThreadStopped?: (threadId: ThreadId) => Effect.Effect; + readonly confirmThreadStopped?: ( + threadId: ThreadId, + /** + * What the session directory keeps of each of the thread's sessions this + * driver ran, in this process or an earlier one: a driver whose own + * evidence a restart lost reads what may still run from these. + */ + known: ReadonlyArray, + ) => Effect.Effect; /** * One session key's runtime is about to stop for a move: note what its @@ -484,6 +492,13 @@ export interface ProviderAdapterShape { readonly streamEvents: Stream.Stream; } +/** One session a driver ran, as the session directory keeps it (confirmThreadStopped). */ +export interface ProviderStoppedSessionHint { + readonly sessionKey: ThreadId; + readonly resumeCursor: unknown; + readonly cwd: string | undefined; +} + /** Whether a thread's agents are proven stopped (ProviderAdapterShape.confirmThreadStopped). */ export type ProviderStopConfirmation = | { readonly confirmed: true } From 4eefc734af8a4e2f1f57c13d263ab97ba03c7769 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 23:34:46 -0500 Subject: [PATCH 26/48] fix(server): a process table that couldn't be read never passes as nothing started When the snapshot at a move's stop failed, the agent's exit watcher read the unknown list as empty and forgot the agent once its group was gone, so confirm passed while a shell in a group of its own lived on. The entry now stays, and confirm says the process list couldn't be read. --- .../src/provider/agentProcessLedger.test.ts | 27 +++++++++++++++++++ .../server/src/provider/agentProcessLedger.ts | 7 ++++- 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/apps/server/src/provider/agentProcessLedger.test.ts b/apps/server/src/provider/agentProcessLedger.test.ts index ed4e3792f..fef2b2cbd 100644 --- a/apps/server/src/provider/agentProcessLedger.test.ts +++ b/apps/server/src/provider/agentProcessLedger.test.ts @@ -1,5 +1,6 @@ import { ThreadId } from "@threadlines/contracts"; import { it } from "@effect/vitest"; +import * as Deferred from "effect/Deferred"; import * as Effect from "effect/Effect"; import { describe, expect } from "vite-plus/test"; @@ -53,4 +54,30 @@ describe.skipIf(process.platform === "win32")("agent process ledger", () => { }), ), ); + + it.live("never reads a process table it couldn't read as nothing started", () => + Effect.scoped( + Effect.gen(function* () { + const ledger = yield* makeAgentProcessLedger({ + settleWait: "50 millis", + probe: { + ...realProcessProbe, + // The agent and its group are gone; what it started can't be known. + exists: () => false, + signal: () => {}, + descendants: () => Effect.succeed(null), + }, + }); + const exited = yield* Deferred.make(); + yield* ledger.track(THREAD, { pid: 424242, exited: Deferred.await(exited) }); + yield* ledger.beforeStop(THREAD); + yield* Deferred.succeed(exited, undefined); + yield* Effect.sleep("20 millis"); + expect(yield* ledger.confirm(THREAD)).toEqual({ + confirmed: false, + reason: "the process list couldn't be read when it stopped", + }); + }), + ), + ); }); diff --git a/apps/server/src/provider/agentProcessLedger.ts b/apps/server/src/provider/agentProcessLedger.ts index 573f65ed7..9ec1d41bc 100644 --- a/apps/server/src/provider/agentProcessLedger.ts +++ b/apps/server/src/provider/agentProcessLedger.ts @@ -174,8 +174,13 @@ export const makeAgentProcessLedger = ( ? probe.exists(-entry.pid) : !entry.leaderGone && probe.exists(entry.pid); const notedAlive = (entry: Entry) => (entry.noted ?? []).filter((pid) => probe.exists(pid)); + // A stop whose process table couldn't be read left nothing known: the + // entry stays, and confirm says so, rather than reading it as nothing. const settled = (entry: Entry) => - entry.leaderGone && !groupAlive(entry) && notedAlive(entry).length === 0; + entry.noted !== null && + entry.leaderGone && + !groupAlive(entry) && + notedAlive(entry).length === 0; const track: AgentProcessLedger["track"] = (sessionKey, agent) => Effect.gen(function* () { From 9bf9593d50d5a208aaf6f89d1abac44ace9d23bb Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 23:35:36 -0500 Subject: [PATCH 27/48] fix(server): a move look begun before a Cancel never stops the user's new work The queue ignored a refused thread.move.quiesce and went on to end the thread's sessions, so cancelling and sending a new message could have that message's session stopped by a look already under way. It now ends sessions only when the projection shows this request quiescing. --- apps/server/src/moves/MoveQueue.test.ts | 33 +++++++++++++++++++++++++ apps/server/src/moves/MoveQueue.ts | 8 ++++++ 2 files changed, 41 insertions(+) diff --git a/apps/server/src/moves/MoveQueue.test.ts b/apps/server/src/moves/MoveQueue.test.ts index f8bb5b30d..23aec6290 100644 --- a/apps/server/src/moves/MoveQueue.test.ts +++ b/apps/server/src/moves/MoveQueue.test.ts @@ -61,10 +61,17 @@ async function harness(options?: { readonly model?: OrchestrationReadModel }) { emptyCommands: 0, /** Runs right before a start, as if a provider turn slipped in. */ beforeStart: null as null | (() => Promise), + /** Runs right before the queue's quiesce lands, as if the user acted first. */ + beforeQuiesce: null as null | (() => Effect.Effect), }; const dispatch = (command: OrchestrationCommand): Effect.Effect => Effect.gen(function* () { + if (command.type === "thread.move.quiesce" && state.beforeQuiesce !== null) { + const first = state.beforeQuiesce; + state.beforeQuiesce = null; + yield* first(); + } const decided = yield* decideOrchestrationCommand({ command, readModel: model }).pipe( Effect.mapError((error) => new ThreadMoveError({ detail: error.message })), ); @@ -185,6 +192,8 @@ async function harness(options?: { readonly model?: OrchestrationReadModel }) { setSession: (status: OrchestrationSession["status"], extra?: Partial) => run(setSession(status, extra)), dispatch: (command: OrchestrationCommand) => run(dispatch(command)), + dispatchEffect: dispatch, + setSessionEffect: setSession, /** The turn ends as a provider ends it: completed, then the session idles. */ endTurn: (turnId: TurnId) => run( @@ -396,6 +405,30 @@ describe("MoveQueue", () => { await h.close(); }); + it("a look begun before a Cancel never stops the work the user started after it", async () => { + const h = await harness(); + await h.startReactor(); + // The user cancels, and sends a new message, while the look is on its way to the quiesce. + h.state.beforeQuiesce = () => + Effect.gen(function* () { + yield* h.dispatchEffect({ + type: "thread.move.unqueue", + commandId: nextCommandId(), + threadId: THREAD, + requestId: h.thread().pendingMove!.requestId, + createdAt: now, + }); + yield* h.setSessionEffect("running", { activeTurnId: TurnId.make("turn-new") }); + }); + await h.queueMove(); + await waitFor(() => (h.thread().pendingMove ?? null) === null, "the cancel"); + await settle(); + expect(h.state.quiesceCalls).toBe(0); + expect(h.thread().session?.status).toBe("running"); + expect(h.thread().move ?? null).toBeNull(); + await h.close(); + }); + it("a Cancel that lost the race to the start cancels the transfer instead", async () => { const h = await harness(); await h.startReactor(); diff --git a/apps/server/src/moves/MoveQueue.ts b/apps/server/src/moves/MoveQueue.ts index bc6c94c0a..93a3d7de7 100644 --- a/apps/server/src/moves/MoveQueue.ts +++ b/apps/server/src/moves/MoveQueue.ts @@ -361,6 +361,14 @@ export const makeMoveQueue = ( }); } } + // Only a move the projection shows quiescing, as this request, ends + // the thread's sessions: a look begun before a Cancel must not stop + // the work the user started after it (its quiesce was refused). + const fenced = (yield* deps.readThread(threadId))?.pendingMove ?? null; + if (fenced === null || fenced.requestId !== requestId || fenced.stage !== "quiescing") { + entry.again = true; + return; + } // Every driver proves its agent's processes are gone. const confirmation = yield* deps.quiesce(threadId); if (!confirmation.confirmed) { From d37284060dcac5ff486c0836ca3af575764d7e2a Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 23:36:15 -0500 Subject: [PATCH 28/48] fix(server): a Cancel that the move's start beats still cancels the transfer If the start committed between the Cancel's read and its unqueue, the frozen thread refused the unqueue and the error ended the Cancel before it could cancel the transfer. A refused unqueue now reads again and cancels the move that started from the same request; it fails only when the queued move is still there. --- apps/server/src/moves/MoveQueue.test.ts | 31 +++++++++++++++++++++++++ apps/server/src/moves/MoveQueue.ts | 23 +++++++++++++----- 2 files changed, 48 insertions(+), 6 deletions(-) diff --git a/apps/server/src/moves/MoveQueue.test.ts b/apps/server/src/moves/MoveQueue.test.ts index 23aec6290..1f95ec482 100644 --- a/apps/server/src/moves/MoveQueue.test.ts +++ b/apps/server/src/moves/MoveQueue.test.ts @@ -63,6 +63,8 @@ async function harness(options?: { readonly model?: OrchestrationReadModel }) { beforeStart: null as null | (() => Promise), /** Runs right before the queue's quiesce lands, as if the user acted first. */ beforeQuiesce: null as null | (() => Effect.Effect), + /** Runs right before a Cancel lands, as if the start won the race. */ + beforeUnqueue: null as null | (() => Effect.Effect), }; const dispatch = (command: OrchestrationCommand): Effect.Effect => @@ -72,6 +74,11 @@ async function harness(options?: { readonly model?: OrchestrationReadModel }) { state.beforeQuiesce = null; yield* first(); } + if (command.type === "thread.move.unqueue" && state.beforeUnqueue !== null) { + const first = state.beforeUnqueue; + state.beforeUnqueue = null; + yield* first(); + } const decided = yield* decideOrchestrationCommand({ command, readModel: model }).pipe( Effect.mapError((error) => new ThreadMoveError({ detail: error.message })), ); @@ -429,6 +436,30 @@ describe("MoveQueue", () => { await h.close(); }); + it("a Cancel the start beats between its read and its landing cancels the transfer", async () => { + const h = await harness(); + // Offline: the move is quiesced and held, ready to start. + h.state.online = false; + await h.startReactor(); + const { requestId } = await h.queueMove(); + await waitFor(() => h.thread().pendingMove?.hold?.kind === "target_unreachable", "the hold"); + h.state.beforeUnqueue = () => + h.dispatchEffect({ + type: "thread.move.start", + commandId: nextCommandId(), + threadId: THREAD, + transferId: ThreadMoveId.make("transfer-raced"), + target: { environmentId: MAC_MINI, label: "Mac mini" }, + agent, + queuedRequestId: requestId, + createdAt: now, + }); + await h.run(h.queue.unqueue({ threadId: THREAD, requestId })); + expect(h.state.cancels).toBe(1); + expect(h.thread().move).toMatchObject({ transferId: "transfer-raced", cancelRequested: true }); + await h.close(); + }); + it("a Cancel that lost the race to the start cancels the transfer instead", async () => { const h = await harness(); await h.startReactor(); diff --git a/apps/server/src/moves/MoveQueue.ts b/apps/server/src/moves/MoveQueue.ts index 93a3d7de7..13c2e67f3 100644 --- a/apps/server/src/moves/MoveQueue.ts +++ b/apps/server/src/moves/MoveQueue.ts @@ -449,8 +449,9 @@ export const makeMoveQueue = ( const read = deps.readThread(input.threadId); const thread = yield* read; if (thread === null) return; + let refused: string | null = null; if (thread.pendingMove?.requestId === input.requestId) { - yield* deps + refused = yield* deps .dispatch({ type: "thread.move.unqueue", commandId: commandId(input.requestId, "unqueue"), @@ -458,12 +459,22 @@ export const makeMoveQueue = ( requestId: input.requestId, createdAt: yield* nowIso, }) - .pipe(Effect.mapError((error) => moveError(error.message))); + .pipe( + Effect.as(null), + Effect.catch((error) => Effect.succeed(error.message)), + ); + } + // Read after the cancel: a start the decider took first (the frozen + // thread refuses the unqueue then) is now the thread's move, and the + // Cancel cancels its transfer instead. + const after = yield* read; + const move = after?.move ?? null; + if (move === null || move.queuedRequestId !== input.requestId) { + if (refused !== null && after?.pendingMove?.requestId === input.requestId) { + return yield* moveError(refused); + } + return; } - // Read after the cancel: a start the decider took first is now the - // thread's move, and the Cancel cancels its transfer instead. - const move = (yield* read)?.move ?? null; - if (move === null || move.queuedRequestId !== input.requestId) return; if (move.movedAway !== undefined) { return yield* moveError(`This thread already moved to ${move.target.label}.`); } From 44e646d69dc5789301a9fd2ec749a14411449e92 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 23:37:05 -0500 Subject: [PATCH 29/48] fix(server): stop and move now says so when a command runs in the user's terminal A suggested command runs in the user's own terminal tab, and the server has no way to stop one there, so Stop and move now sat on a command-run wait with nothing to say why. It is now refused with words: stop the command there, or move the thread when it's done. Move when it's done still waits for it. --- .../orchestration/decider.threadMoves.test.ts | 24 +++++++++++++++++++ .../src/orchestration/threadMoveDecisions.ts | 7 ++++++ 2 files changed, 31 insertions(+) diff --git a/apps/server/src/orchestration/decider.threadMoves.test.ts b/apps/server/src/orchestration/decider.threadMoves.test.ts index 16639dac5..d891dcc2a 100644 --- a/apps/server/src/orchestration/decider.threadMoves.test.ts +++ b/apps/server/src/orchestration/decider.threadMoves.test.ts @@ -545,6 +545,30 @@ describe("a move that waits for the thread to be free", () => { expect(threadOf(model).pendingMove?.stage).toBe("quiescing"); }); + it("stop and move now is refused while a command runs in the user's terminal", async () => { + const running = await seeded(); + const withRun = { + ...running, + threads: running.threads.map((thread) => + thread.id === THREAD + ? ({ + ...thread, + commandRuns: [{ runId: "run-1", status: "running", source: { kind: "user" } }], + } as unknown as typeof thread) + : thread, + ), + }; + expect((await apply(withRun, queue("stop-now"))).refusal).toContain( + "A command from this thread is running in your terminal. Stop it there, or move the thread when it's done.", + ); + // Move when it's done waits for it. + const { model, refusal } = await apply(withRun, queue("when-done")); + expect(refusal).toBeNull(); + expect(threadOf(model).pendingMove?.waitsFor?.map((wait) => wait.code)).toEqual([ + "command_run", + ]); + }); + it("a cancel or an end naming an older request changes nothing", async () => { let { model } = await apply(await seeded(), queue()); const old = ThreadMoveRequestId.make("request-old"); diff --git a/apps/server/src/orchestration/threadMoveDecisions.ts b/apps/server/src/orchestration/threadMoveDecisions.ts index 0f6b236cc..c437d23bd 100644 --- a/apps/server/src/orchestration/threadMoveDecisions.ts +++ b/apps/server/src/orchestration/threadMoveDecisions.ts @@ -91,6 +91,8 @@ const WAIT_WORDS: Record = { }; const BUSY_WORDS = "Wait for this thread to finish what it's doing, then move it."; +const STOP_NOW_COMMAND_RUN_WORDS = + "A command from this thread is running in your terminal. Stop it there, or move the thread when it's done."; const SHARED_CHECKOUT_WORDS = "Another thread is working in the same folder. Wait for it to finish, then move this one."; @@ -585,6 +587,11 @@ export function decideMoveQueue( const refusals = threadMoveRefusals(readModel, thread); const block = firstMoveBlock(refusals); if (block !== null) return refuse(block.words); + // A command runs in the user's own terminal tab, which the server doesn't + // stop: "Stop and move now" would wait on it with nothing to show why. + if (command.mode === "stop-now" && refusals.some((entry) => entry.code === "command_run")) { + return refuse(STOP_NOW_COMMAND_RUN_WORDS); + } const queued: PlannedEvent = { ...baseFor(thread.id), type: "thread.move-queued", From 7c2bea91029909b57e50bef5391a230a349eb041 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Mon, 16 Nov 2026 00:30:59 -0500 Subject: [PATCH 30/48] fix(server): a stop that finds its session already gone counts as stopped A when-done move's quiesce and the reactor's thread.session-stop-requested both stop the same session. The one that lost the race got the driver's "Unknown adapter thread" (ACP) or "thread is closed" (Claude) error, and the reactor's handler stopped there, skipping room agents, early starts, the side answer and the outside-work bookkeeping. ProviderService's stop now reads either error as nothing left to stop, for every caller. --- .../provider/Layers/ProviderService.test.ts | 27 +++++++++++++++++++ .../src/provider/Layers/ProviderService.ts | 10 ++++++- 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/apps/server/src/provider/Layers/ProviderService.test.ts b/apps/server/src/provider/Layers/ProviderService.test.ts index 01c916de7..c7c65a6e7 100644 --- a/apps/server/src/provider/Layers/ProviderService.test.ts +++ b/apps/server/src/provider/Layers/ProviderService.test.ts @@ -47,6 +47,7 @@ import * as SqlClient from "effect/sql/SqlClient"; import { ProviderAdapterProcessError, ProviderAdapterRequestError, + ProviderAdapterSessionClosedError, ProviderAdapterSessionNotFoundError, ProviderUnsupportedError, ProviderValidationError, @@ -3245,4 +3246,30 @@ fence.layer("ProviderServiceLive moving threads", (it) => { fence.fenced.delete(RECOVERING); }), ); + + it.effect("a stop that finds its session already stopped by another counts as stopped", () => + Effect.gen(function* () { + const provider = yield* ProviderService; + for (const gone of [ + (threadId: ThreadId) => + new ProviderAdapterSessionNotFoundError({ provider: "codex", threadId }), + (threadId: ThreadId) => + new ProviderAdapterSessionClosedError({ provider: "codex", threadId }), + ]) { + const RACED = asThreadId("3d5f7b9c-2e4a-4c6e-8f0a-1b3c5d7e9f2a"); + yield* startOn(codexInstanceId, RACED); + // A move's quiesce and the reactor's stop race: the other one got there first. + const stopNormally = fence.codex.stopSession.getMockImplementation()!; + fence.codex.stopSession.mockImplementationOnce((threadId) => + stopNormally(threadId).pipe(Effect.andThen(Effect.fail(gone(threadId)))), + ); + const stopped = yield* provider.stopSession({ threadId: RACED }).pipe(Effect.result); + assert.equal(stopped._tag, "Success"); + const left = (yield* provider.listSessions()).filter( + (session) => session.threadId === RACED, + ); + assert.deepEqual(left, []); + } + }), + ); }); diff --git a/apps/server/src/provider/Layers/ProviderService.ts b/apps/server/src/provider/Layers/ProviderService.ts index a0d958eb5..dd520a327 100644 --- a/apps/server/src/provider/Layers/ProviderService.ts +++ b/apps/server/src/provider/Layers/ProviderService.ts @@ -2600,7 +2600,15 @@ const makeProviderService = Effect.fn("makeProviderService")(function* ( ) { yield* routed.adapter.noteBeforeStop(routed.threadId); } - yield* routed.adapter.stopSession(routed.threadId); + // Two stops of one session can race (a move's quiesce and the + // reactor's session stop): the one that finds it already gone, or + // closed, has nothing left to stop. + yield* routed.adapter.stopSession(routed.threadId).pipe( + Effect.catchTags({ + ProviderAdapterSessionNotFoundError: () => Effect.void, + ProviderAdapterSessionClosedError: () => Effect.void, + }), + ); } // A side answer's runtime is disposable: nothing ever resumes it, so // its binding goes with it rather than piling up as "stopped". From 86037ff2811f4700f64c3b3fa3b668928ee612f6 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:39:35 -0500 Subject: [PATCH 31/48] feat(web): move a working thread when it's done, or stop it and move now The move dialog offers a busy thread two ways to move and queues the one picked (moves.queue); a computer out of reach can be picked too, and the move waits for it. While a move waits, the notice above the message box says when it goes or what holds it, with Cancel and Try again, and the send button says why nothing new can go. A queued move that couldn't start says why until dismissed. The sidebar row reads "move queued" once the thread is only waiting to go, and the store follows the four queued-move events live. --- .../components/ChatView.browser-harness.tsx | 1 + .../ChatView.move-thread.browser.tsx | 260 ++++++++++++++++++ apps/web/src/components/ChatView.tsx | 56 +++- .../src/components/CommandPaletteDialog.tsx | 2 +- apps/web/src/components/ThreadMoveDialog.tsx | 3 +- .../components/ThreadMoveDialogContent.tsx | 170 ++++++++++-- apps/web/src/components/chat/ChatComposer.tsx | 11 + .../chat/ComposerPrimaryActions.tsx | 102 +++++-- .../src/components/chat/threadMoveNotice.tsx | 203 +++++++++++++- apps/web/src/components/sidebar/InboxRows.tsx | 24 +- apps/web/src/rpc/wsRpcClient.ts | 10 + apps/web/src/session-logic.ts | 15 +- apps/web/src/store.test.ts | 115 ++++++++ apps/web/src/store.ts | 114 +++++++- apps/web/src/threadMove.test.ts | 50 ++++ apps/web/src/threadMove.ts | 144 +++++++++- apps/web/src/types.ts | 18 +- 17 files changed, 1205 insertions(+), 93 deletions(-) diff --git a/apps/web/src/components/ChatView.browser-harness.tsx b/apps/web/src/components/ChatView.browser-harness.tsx index 2b50911bc..5fce347e6 100644 --- a/apps/web/src/components/ChatView.browser-harness.tsx +++ b/apps/web/src/components/ChatView.browser-harness.tsx @@ -888,6 +888,7 @@ function toShellThread(thread: OrchestrationReadModel["threads"][number]) { hasPendingUserInput: false, hasActionableProposedPlan: false, ...(thread.move ? { move: thread.move } : {}), + ...(thread.pendingMove ? { pendingMove: thread.pendingMove } : {}), ...(thread.movedFrom ? { movedFrom: thread.movedFrom } : {}), ...(thread.outsideWork ? { outsideWork: thread.outsideWork } : {}), }; diff --git a/apps/web/src/components/ChatView.move-thread.browser.tsx b/apps/web/src/components/ChatView.move-thread.browser.tsx index 29d858aa7..9bfd573ed 100644 --- a/apps/web/src/components/ChatView.move-thread.browser.tsx +++ b/apps/web/src/components/ChatView.move-thread.browser.tsx @@ -12,6 +12,8 @@ import { ThreadMoveError, ThreadMoveId, type ThreadMovePreview, + ThreadMoveRequestId, + type ThreadPendingMove, WS_METHODS, } from "@threadlines/contracts"; import { page, userEvent } from "vite-plus/test/browser"; @@ -21,10 +23,12 @@ import { __setComputerLinksClientForTests, useComputerLinksStore, } from "../environments/links/computerLinks"; +import { useComposerDraftStore } from "../composerDraftStore"; import { __resetThreadMovesForTests, __setThreadMovesClientForTests, readThreadMoveOffered, + type ThreadMovesClient, } from "../environments/links/threadMoves"; import { __setProjectSetupsClientForTests } from "../environments/projectSetups/projectSetups"; import { useSavedEnvironmentRuntimeStore } from "../environments/runtime"; @@ -32,6 +36,7 @@ import { type NormalizedWsRpcRequestBody } from "../../test/wsRpcHarness"; import { THREAD_ID, + THREAD_REF, THREAD_TITLE, LOCAL_ENVIRONMENT_ID, NOW_ISO, @@ -158,6 +163,55 @@ const FLEET_MOVE_PREVIEW: ThreadMovePreview = { ], }; +/** The calls of a waiting move (Part 3b), for a stand-in client whose test never queues. */ +const QUEUE_CALLS_UNUSED: Pick = { + queue: async () => { + throw new Error("This test doesn't queue a move."); + }, + unqueue: async () => undefined, + retry: async () => undefined, +}; + +const QUEUED_REQUEST_ID = ThreadMoveRequestId.make("queued-browser"); + +/** The test thread's move to Mac mini, waiting for its turn to end. */ +function fleetPendingMove(overrides: Partial = {}): ThreadPendingMove { + return { + requestId: QUEUED_REQUEST_ID, + target: { environmentId: FLEET_OFFLINE_ENVIRONMENT_ID, label: "Mac mini" }, + agent: { instanceId: ProviderInstanceId.make("codex"), model: "gpt-5" }, + mode: "when-done", + stage: "waiting", + requestedAt: NOW_ISO, + requestedBy: { kind: "user" }, + waitsFor: [{ code: "turn", words: "when this turn ends" }], + ...overrides, + }; +} + +/** One event on the open thread's detail stream, after `sequence - 1` others. */ +function emitThreadEvent( + sequence: number, + event: Pick & { readonly eventId: string }, +): void { + rpcHarness.emitStreamValue(ORCHESTRATION_WS_METHODS.subscribeThread, { + kind: "event", + event: { + sequence, + eventId: EventId.make(event.eventId), + aggregateKind: "thread", + aggregateId: THREAD_ID, + occurredAt: NOW_ISO, + commandId: null, + causationEventId: null, + correlationId: null, + metadata: {}, + type: event.type, + payload: event.payload, + } as OrchestrationEvent, + }); +} + /** * Saves Mac mini, where these moves go. Saving a computer is also what starts * this app's links streams here: the test server only says it takes links @@ -355,6 +409,7 @@ describe("ChatView (full app): moving a thread to another computer", () => { preview: async () => FLEET_MOVE_PREVIEW, start: refusedStart, cancel: async () => undefined, + ...QUEUE_CALLS_UNUSED, }); await dialog.getByTestId("thread-move-confirm").click(); await expect @@ -378,6 +433,210 @@ describe("ChatView (full app): moving a thread to another computer", () => { } }); + it("offers a busy thread two ways to move, and queues the one picked", async () => { + setPrimaryLinksSnapshot( + fleetLinksSnapshot({ + outbound: [fleetOutboundLink(FLEET_OFFLINE_ENVIRONMENT_ID, "Mac mini")], + }), + ); + const macMini = FLEET_MOVE_PREVIEW.targets[1]!; + const queues: Array = []; + const mounted = await mountChatView({ + viewport: WIDE_FOOTER_VIEWPORT, + snapshot: createSnapshotForTargetUser({ + targetMessageId: "msg-user-move-busy" as MessageId, + targetText: "move busy", + }), + configureFixture: withThreadMoveFixture, + resolveRpc: (body) => { + if (body._tag === WS_METHODS.movesPreview) { + return { + ...FLEET_MOVE_PREVIEW, + waitsFor: [ + { code: "turn", words: "when this turn ends" }, + { code: "background", words: "when its background work finishes" }, + ], + targets: [macMini], + } satisfies ThreadMovePreview; + } + if (body._tag === WS_METHODS.movesQueue) { + queues.push(body); + return { requestId: QUEUED_REQUEST_ID }; + } + return undefined; + }, + }); + + try { + await waitForServerConfigToApply(); + await saveMacMini("connected"); + await vi.waitFor(() => { + expect(readThreadMoveOffered(LOCAL_ENVIRONMENT_ID, undefined)).toBe(true); + }); + await openCommandPaletteFromTrigger(); + await page + .getByTestId("command-palette") + .getByText("Move to another computer", { exact: true }) + .click(); + const dialog = page.getByTestId("thread-move-dialog"); + // What it's busy with, then the two ways: when it's done comes picked. + await expect + .element(dialog.getByTestId("thread-move-busy")) + .toHaveTextContent( + "This thread is busy: its agent is working on a turn and background work is still running.", + ); + await expect + .element(dialog.getByTestId("thread-move-mode-when-done")) + .toMatchTextContent("Move when it's doneMoves to Mac mini when this turn ends."); + await expect + .element(dialog.getByRole("radio", { name: /Move when it's done/u })) + .toBeChecked(); + await expect + .element(dialog.getByTestId("thread-move-confirm")) + .toHaveTextContent("Move when done"); + + await dialog.getByTestId("thread-move-mode-stop-now").click(); + await expect + .element(dialog.getByTestId("thread-move-confirm")) + .toHaveTextContent("Stop and move"); + await dialog.getByTestId("thread-move-confirm").click(); + await expect.element(page.getByTestId("thread-move-dialog")).not.toBeInTheDocument(); + expect(queues).toEqual([ + expect.objectContaining({ + threadId: THREAD_ID, + targetEnvironmentId: FLEET_OFFLINE_ENVIRONMENT_ID, + agent: { instanceId: "codex", model: "gpt-5" }, + mode: "stop-now", + }), + ]); + } finally { + await mounted.cleanup(); + } + }); + + it("says when a waiting move goes, holds new messages, and cancels the move it shows", async () => { + setPrimaryLinksSnapshot( + fleetLinksSnapshot({ + outbound: [ + fleetOutboundLink(FLEET_OFFLINE_ENVIRONMENT_ID, "Mac mini", { connection: "connected" }), + ], + }), + ); + const calls: Array = []; + const mounted = await mountChatView({ + viewport: WIDE_FOOTER_VIEWPORT, + snapshot: withTestThread( + createSnapshotForTargetUser({ + targetMessageId: "msg-user-move-waits" as MessageId, + targetText: "move waits", + }), + { pendingMove: fleetPendingMove() }, + ), + configureFixture: withThreadMoveFixture, + resolveRpc: (body) => { + if (body._tag === WS_METHODS.movesUnqueue || body._tag === WS_METHODS.movesRetry) { + calls.push(body); + return null; + } + return undefined; + }, + }); + + try { + await waitForServerConfigToApply(); + await saveMacMini("connected"); + await waitForComposerNotice("Moves to Mac mini when this turn ends."); + await expect + .element(page.getByTestId(`thread-meta-${THREAD_ID}`)) + .toMatchTextContent("move queued"); + + // The box still takes typing, but the message can't go, and says why. + useComposerDraftStore.getState().setPrompt(THREAD_REF, "one more thing"); + const reason = + "This thread is waiting to move to Mac mini. Cancel the move to keep working here."; + await expect.element(page.getByRole("button", { name: reason })).toBeDisabled(); + + // The agent can't be proven stopped: it says so, with Try again. + emitThreadEvent(2, { + eventId: "move-queue-held", + type: "thread.move-queue-status-set", + payload: { + threadId: THREAD_ID, + requestId: QUEUED_REQUEST_ID, + waitsFor: [], + hold: { + kind: "stop_unconfirmed", + words: "Couldn't confirm the agent stopped: Codex didn't exit.", + }, + updatedAt: NOW_ISO, + }, + }); + await waitForComposerNotice("Couldn't confirm the agent stopped: Codex didn't exit."); + await page.getByRole("button", { name: "Try again" }).click(); + await page.getByRole("button", { name: "Cancel the move", exact: true }).click(); + await vi.waitFor(() => { + expect(calls).toEqual([ + expect.objectContaining({ + _tag: WS_METHODS.movesRetry, + threadId: THREAD_ID, + requestId: QUEUED_REQUEST_ID, + }), + expect.objectContaining({ + _tag: WS_METHODS.movesUnqueue, + threadId: THREAD_ID, + requestId: QUEUED_REQUEST_ID, + }), + ]); + }); + + // It couldn't go after all: why, until dismissed, and the box sends again. + emitThreadEvent(3, { + eventId: "move-queue-ended", + type: "thread.move-queue-ended", + payload: { + threadId: THREAD_ID, + requestId: QUEUED_REQUEST_ID, + reason: "blocked", + words: + "this thread's agent set itself a time to come back. Cancel it first, then move again.", + endedAt: NOW_ISO, + }, + }); + emitThreadEvent(4, { + eventId: "move-not-started-activity", + type: "thread.activity-appended", + payload: { + threadId: THREAD_ID, + activity: { + id: EventId.make(`move-not-started:${QUEUED_REQUEST_ID}`), + tone: "error", + kind: "thread.move-not-started", + summary: "Didn't move to Mac mini", + payload: { + requestId: QUEUED_REQUEST_ID, + targetLabel: "Mac mini", + reason: "blocked", + words: + "this thread's agent set itself a time to come back. Cancel it first, then move again.", + }, + turnId: null, + createdAt: NOW_ISO, + }, + }, + }); + await waitForComposerNotice( + "Didn't move: this thread's agent set itself a time to come back. Cancel it first, then move again.", + ); + await expect.element(page.getByRole("button", { name: "Send message" })).toBeEnabled(); + await page.getByRole("button", { name: "Dismiss why it didn't move to Mac mini" }).click(); + await expect + .element(page.getByRole("button", { name: "Dismiss why it didn't move to Mac mini" })) + .not.toBeInTheDocument(); + } finally { + await mounted.cleanup(); + } + }); + it("sets up a computer that lacks the project from the move, then offers it", async () => { setPrimaryLinksSnapshot( fleetLinksSnapshot({ @@ -476,6 +735,7 @@ describe("ChatView (full app): moving a thread to another computer", () => { preview, start: async () => ({ transferId: ThreadMoveId.make("move-after-setup") }), cancel: async () => undefined, + ...QUEUE_CALLS_UNUSED, }); mac.push([ projectSetupJob({ diff --git a/apps/web/src/components/ChatView.tsx b/apps/web/src/components/ChatView.tsx index eac42c480..743521f89 100644 --- a/apps/web/src/components/ChatView.tsx +++ b/apps/web/src/components/ChatView.tsx @@ -347,12 +347,16 @@ import { type UsageLimitNoticeState, } from "./chat/threadErrorNotice"; import { type ComposerNotice, selectComposerNotices } from "./chat/composerNotices"; -import { useThreadMoveNotice } from "./chat/threadMoveNotice"; +import { + useMoveNotStartedNotice, + usePendingMoveNotice, + useThreadMoveNotice, +} from "./chat/threadMoveNotice"; import { useOutsideWorkNotice } from "./chat/outsideWorkNotice"; import { openProjectSetupDialog } from "./ProjectSetupDialog"; import { useProjectSetupTargets } from "../environments/projectSetups/projectSetupTargets"; import { useComputerLabel } from "./sidebar/linkedComputers"; -import { threadMoveLockReason } from "../threadMove"; +import { pendingMoveSendBlockReason, threadMoveLockReason } from "../threadMove"; import { MAX_HIDDEN_MOUNTED_TERMINAL_THREADS, DEFAULT_SCROLL_END_TOLERANCE_PX, @@ -2896,6 +2900,19 @@ export default function ChatView(props: ChatViewProps) { () => (threadMoveLockText === null ? null : { reason: threadMoveLockText }), [threadMoveLockText], ); + // A move waiting for the thread to be free (Part 3b): the box still takes + // typing and answers, but nothing that starts new work goes. + const pendingMoveSendBlockText = activeThread?.pendingMove + ? pendingMoveSendBlockReason(activeThread.pendingMove) + : null; + const pendingMoveSendBlock = useMemo( + () => + // A reply that answers the agent's questions still goes: the turn needs it. + pendingMoveSendBlockText === null || replyingInChatTo !== null + ? null + : { reason: pendingMoveSendBlockText }, + [pendingMoveSendBlockText, replyingInChatTo], + ); const revertTurnCountByUserMessageId = useMemo( () => deriveThreadRevertTurnCountByUserMessageId( @@ -5536,6 +5553,18 @@ export default function ChatView(props: ChatViewProps) { } return; } + // A move waits for the thread to be free (Part 3b): nothing that starts + // work goes, and the draft stays. The send button says why; Enter too. + if (activeThread.pendingMove) { + toastManager.add( + stackedThreadToast({ + type: "info", + title: "This thread is waiting to move", + description: pendingMoveSendBlockReason(activeThread.pendingMove), + }), + ); + return; + } // A message for the agent whose turn waits on approvals answers them // first. Only the draft that was sent goes out afterwards, to the agent it // was for: one edited, or readdressed, meanwhile stays in the box, and its @@ -7329,6 +7358,8 @@ export default function ChatView(props: ChatViewProps) { thread: activeThread, agentName: outsideWorkAgentName, }); + // A queued move that ended without starting says why until dismissed. + const moveNotStartedNotice = useMoveNotStartedNotice({ thread: activeThread }); // While a reply in chat answers the questions, the box says so, with the // way back to the card. const replyInChatNotice = useMemo( @@ -7365,11 +7396,13 @@ export default function ChatView(props: ChatViewProps) { sessionStartupNotice, removedAccountNotice, outsideWorkNotice, + moveNotStartedNotice, ...infrastructureComposerNotices, ]), [ checkoutMissingNotice, infrastructureComposerNotices, + moveNotStartedNotice, outsideWorkNotice, providerStatusNotice, removedAccountNotice, @@ -8002,9 +8035,23 @@ export default function ChatView(props: ChatViewProps) { computerLabel: activeThreadComputerLabel, onContinueHere: continueHereAsNewThread, }); + // A move that waits for the thread to be free (Part 3b) leads the dock: it + // is why nothing new can be sent. A reply to questions, which can still + // go, keeps its own notice in front. + const pendingMoveNotice = usePendingMoveNotice({ + thread: activeThread, + computerLabel: activeThreadComputerLabel, + }); const dockedComposerNotices = useMemo( - () => (threadMoveNotice ? [threadMoveNotice] : composerNotices), - [composerNotices, threadMoveNotice], + () => + threadMoveNotice + ? [threadMoveNotice] + : pendingMoveNotice === null + ? composerNotices + : replyingInChatTo !== null + ? [...composerNotices, pendingMoveNotice] + : [pendingMoveNotice, ...composerNotices], + [composerNotices, pendingMoveNotice, replyingInChatTo, threadMoveNotice], ); const updateForkDialogInstruction = useCallback((instruction: string) => { @@ -8913,6 +8960,7 @@ export default function ChatView(props: ChatViewProps) { isPreparingWorktree={isPreparingWorktree} environmentUnavailable={activeEnvironmentUnavailableState} threadLocked={threadMoveComposerLock} + sendBlocked={pendingMoveSendBlock} activePendingApproval={activePendingApproval} pendingApprovals={pendingApprovals} pendingUserInputs={composerPendingUserInputs} diff --git a/apps/web/src/components/CommandPaletteDialog.tsx b/apps/web/src/components/CommandPaletteDialog.tsx index c1e9f92a6..05877d732 100644 --- a/apps/web/src/components/CommandPaletteDialog.tsx +++ b/apps/web/src/components/CommandPaletteDialog.tsx @@ -592,7 +592,7 @@ export function OpenCommandPaletteDialog() { const projectSetupTargets = useProjectSetupTargets(openThreadProjectRef); const activeThreadMoveOffered = useThreadMoveOffered( activeThread?.environmentId ?? null, - threadMoveStatus(activeThread?.move), + activeThread ? threadMoveStatus(activeThread) : undefined, ); // The open thread's own actions are its menu's (useThreadMenu); the inbox // says whether it offers Wrap up or Reopen. This mounts only while open. diff --git a/apps/web/src/components/ThreadMoveDialog.tsx b/apps/web/src/components/ThreadMoveDialog.tsx index 087653b22..63094e56d 100644 --- a/apps/web/src/components/ThreadMoveDialog.tsx +++ b/apps/web/src/components/ThreadMoveDialog.tsx @@ -3,7 +3,8 @@ * every entry point (the sidebar's thread menu, the command palette, the chat * header). Each asks through openThreadMoveDialog; the host mounted once at * the root shows it. The dialog asks the thread's computer where the thread - * could go and what would go with it, then starts the move there. + * could go and what would go with it, then starts the move there, or, for a + * busy thread or a computer out of reach, queues one that waits (Part 3b). */ import type { ScopedThreadRef } from "@threadlines/contracts"; import { Suspense } from "react"; diff --git a/apps/web/src/components/ThreadMoveDialogContent.tsx b/apps/web/src/components/ThreadMoveDialogContent.tsx index 2383504f8..bcf50d35f 100644 --- a/apps/web/src/components/ThreadMoveDialogContent.tsx +++ b/apps/web/src/components/ThreadMoveDialogContent.tsx @@ -5,10 +5,13 @@ */ import type { EnvironmentId, + ModelSelection, ScopedProjectRef, ScopedThreadRef, ThreadMovePreview, ThreadMoveTargetPreview, + ThreadMoveWait, + ThreadPendingMoveMode, } from "@threadlines/contracts"; import { useNavigate } from "@tanstack/react-router"; import { useEffect, useMemo, useRef, useState } from "react"; @@ -21,7 +24,10 @@ import { } from "../environments/projectSetups/projectSetupTargets"; import { selectSidebarThreadSummaryByRef, useStore } from "../store"; import { + describeThreadMoveWaits, + describeThreadMoveWhenDone, formatThreadMoveChanges, + isThreadMoveTargetPickable, pickDefaultMoveAgent, pickDefaultMoveTarget, threadMoveAgentKey, @@ -31,6 +37,7 @@ import { import { openProjectSetupDialog } from "./ProjectSetupDialog"; import { useComputerLabel } from "./sidebar/linkedComputers"; import { Button } from "./ui/button"; +import { Radio, RadioGroup } from "./ui/radio-group"; import { DialogDescription, DialogFooter, @@ -64,10 +71,18 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread const projectId = useStore( (state) => selectSidebarThreadSummaryByRef(state, threadRef)?.projectId ?? null, ); + // The agent a move to a computer that can't be reached asks for: the + // thread's own, since that computer can't say which it has. + const threadModelSelection = useStore( + (state) => + state.environmentStateById[threadRef.environmentId]?.threadShellById[threadRef.threadId] + ?.modelSelection ?? null, + ); const computerLabel = useComputerLabel()(threadRef.environmentId, threadRef.environmentId); const [previewState, setPreviewState] = useState({ status: "loading" }); const [targetId, setTargetId] = useState(null); const [agent, setAgent] = useState(null); + const [mode, setMode] = useState("when-done"); const [starting, setStarting] = useState(false); const [startError, setStartError] = useState(null); // Asked again when a computer that lacked the project has been set up. @@ -91,7 +106,7 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread preview.targets.find( (candidate) => candidate.environmentId === pickedTargetRef.current && - candidate.refusal === undefined, + isThreadMoveTargetPickable(candidate), ) ?? pickDefaultMoveTarget(preview.targets); setTargetId(target?.environmentId ?? null); setAgent(target ? pickDefaultMoveAgent(target) : null); @@ -117,24 +132,46 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread const preview = previewState.status === "ready" ? previewState.preview : null; const target = preview?.targets.find((candidate) => candidate.environmentId === targetId) ?? null; + // Busy: the move can wait for the thread to be free, or stop it first. + const waits = preview?.waitsFor ?? []; + const busy = waits.length > 0; + // Out of reach: a move to it waits for it to connect. + const unreachable = target?.refusalCode === "unreachable"; + // Only a server that queues moves says either, so asking it to queue is + // safe; anything else starts the move at once, as every server does. + const queues = busy || unreachable; + const agentSelection: ModelSelection | null = + target === null + ? null + : unreachable + ? threadModelSelection + : agent !== null + ? threadMoveAgentSelection(target, agent) + : null; const canMove = preview !== null && preview.refusal === undefined && target !== null && - target.refusal === undefined && - agent !== null && + isThreadMoveTargetPickable(target) && + agentSelection !== null && !starting; const start = async () => { - if (!canMove || target === null || agent === null) return; + if (!canMove || target === null || agentSelection === null) return; setStarting(true); setStartError(null); try { - await requireThreadMovesClient(threadRef.environmentId, computerLabel).start({ + const client = requireThreadMovesClient(threadRef.environmentId, computerLabel); + const input = { threadId: threadRef.threadId, targetEnvironmentId: target.environmentId, - agent: threadMoveAgentSelection(target, agent), - }); + agent: agentSelection, + }; + if (queues) { + await client.queue({ ...input, mode: busy ? mode : "when-done" }); + } else { + await client.start(input); + } closeThreadMoveDialog(); } catch (error) { setStartError(describeComputerLinkFailure(error, `${target.label} couldn't take it.`)); @@ -182,7 +219,7 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread setStartError(null); }} /> - {target?.refusal !== undefined ? ( + {target?.refusal !== undefined && !unreachable ? (

{target.refusal} @@ -241,15 +278,34 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread Your copy here stays, read-only.

- { - setAgent(next); - setStartError(null); - }} - /> + {unreachable ? ( +

+ {target.refusal} The move waits for it to connect, and keeps the agent this + thread uses here. +

+ ) : ( + { + setAgent(next); + setStartError(null); + }} + /> + )} + {busy ? ( + + ) : null} {target.access === "ask" ? (

{target.label} asks its user first. @@ -274,13 +330,84 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread Cancel ); } +/** + * A busy thread's two ways to move (Part 3b): once it is free, or now, after + * its agent session ends. Says first what it is busy with. + */ +function MoveWhenField({ + label, + waits, + mode, + disabled, + onChange, +}: { + readonly label: string; + readonly waits: ReadonlyArray; + readonly mode: ThreadPendingMoveMode; + readonly disabled: boolean; + readonly onChange: (mode: ThreadPendingMoveMode) => void; +}) { + const choices: ReadonlyArray<{ + readonly value: ThreadPendingMoveMode; + readonly label: string; + readonly detail: string; + }> = [ + { + value: "when-done", + label: "Move when it's done", + detail: describeThreadMoveWhenDone(label, waits), + }, + { + value: "stop-now", + label: "Stop and move now", + detail: "Stops the agent and anything it left running, then moves.", + }, + ]; + return ( +

+

+ {describeThreadMoveWaits(waits)} +

+ onChange(value as ThreadPendingMoveMode)} + aria-label="When it moves" + className="gap-0.5" + > + {choices.map((choice) => ( + + ))} + +
+ ); +} + /** The computers the thread can go to; one that can't take it is listed, disabled, with why. */ function TargetField({ targets, @@ -312,8 +439,11 @@ function TargetField({ diff --git a/apps/web/src/components/chat/ChatComposer.tsx b/apps/web/src/components/chat/ChatComposer.tsx index 3821780cd..ad471638c 100644 --- a/apps/web/src/components/chat/ChatComposer.tsx +++ b/apps/web/src/components/chat/ChatComposer.tsx @@ -347,6 +347,7 @@ const ComposerFooterPrimaryActions = memo(function ComposerFooterPrimaryActions( isSendBusy: boolean; isConnecting: boolean; isEnvironmentUnavailable: boolean; + sendBlockedReason: string | null; hasSendableContent: boolean; preserveComposerFocusOnPointerDown?: boolean; runtimeMode: RuntimeMode; @@ -395,6 +396,7 @@ const ComposerFooterPrimaryActions = memo(function ComposerFooterPrimaryActions( isSendBusy={props.isSendBusy} isConnecting={props.isConnecting} isEnvironmentUnavailable={props.isEnvironmentUnavailable} + sendBlockedReason={props.sendBlockedReason} isPreparingWorktree={props.isPreparingWorktree} hasSendableContent={props.hasSendableContent} preserveComposerFocusOnPointerDown={props.preserveComposerFocusOnPointerDown ?? false} @@ -503,6 +505,12 @@ export interface ChatComposerProps { * computer (docs/design/fleet.md, Part 3). `reason` is the placeholder. */ threadLocked?: { readonly reason: string } | null; + /** + * Nothing that starts new work can be sent: a move waits for the thread to + * be free (Part 3b). Typing, approvals and stopping still work; `reason` + * is what the send button says. + */ + sendBlocked?: { readonly reason: string } | null; // Pending approvals / inputs activePendingApproval: PendingApproval | null; @@ -672,6 +680,7 @@ export const ChatComposer = memo(function ChatComposer(props: ChatComposerProps) isPreparingWorktree, environmentUnavailable, threadLocked = null, + sendBlocked = null, activePendingApproval, pendingApprovals, pendingUserInputs, @@ -3505,6 +3514,7 @@ export const ChatComposer = memo(function ChatComposer(props: ChatComposerProps) isSendBusy || isConnecting || composerUnavailable || + sendBlocked !== null || (activePendingApproval !== null && respondingRequestIds.includes(activePendingApproval.requestId)); const approvalSendButton = @@ -4456,6 +4466,7 @@ export const ChatComposer = memo(function ChatComposer(props: ChatComposerProps) isSendBusy={isSendBusy} isConnecting={isConnecting} isEnvironmentUnavailable={composerUnavailable} + sendBlockedReason={sendBlocked?.reason ?? null} isPreparingWorktree={isPreparingWorktree} hasSendableContent={ !hasBlockingQuestion && composerSendState.hasSendableContent diff --git a/apps/web/src/components/chat/ComposerPrimaryActions.tsx b/apps/web/src/components/chat/ComposerPrimaryActions.tsx index b7f87c625..a02c4fb7e 100644 --- a/apps/web/src/components/chat/ComposerPrimaryActions.tsx +++ b/apps/web/src/components/chat/ComposerPrimaryActions.tsx @@ -31,6 +31,11 @@ interface ComposerPrimaryActionsProps { isSendBusy: boolean; isConnecting: boolean; isEnvironmentUnavailable: boolean; + /** + * Why nothing that starts work can be sent now (a move waits, Part 3b); the + * send button says it. Stop still works. Absent or null: nothing blocks. + */ + sendBlockedReason?: string | null; isPreparingWorktree: boolean; hasSendableContent: boolean; preserveComposerFocusOnPointerDown?: boolean; @@ -371,6 +376,7 @@ export const ComposerPrimaryActions = memo(function ComposerPrimaryActions({ isSendBusy, isConnecting, isEnvironmentUnavailable, + sendBlockedReason = null, isPreparingWorktree, hasSendableContent, preserveComposerFocusOnPointerDown = false, @@ -388,6 +394,13 @@ export const ComposerPrimaryActions = memo(function ComposerPrimaryActions({ const pointerFocusProps = preserveComposerFocusOnPointerDown ? { onPointerDown: preventPointerFocus } : undefined; + const sendUnavailable = + isSendBusy || isConnecting || isEnvironmentUnavailable || sendBlockedReason !== null; + + // Something typed that can't go: one send button, saying why on hover. + if (sendBlockedReason !== null && hasSendableContent) { + return ; + } // Also while the agent at work only waits on background work, when there // is no turn to steer or stop. @@ -397,7 +410,7 @@ export const ComposerPrimaryActions = memo(function ComposerPrimaryActions({ @@ -436,7 +449,7 @@ export const ComposerPrimaryActions = memo(function ComposerPrimaryActions({ }, ]} onChange={onFollowUpDeliveryChange} - disabled={isSendBusy || isConnecting || isEnvironmentUnavailable} + disabled={sendUnavailable} pointerFocusProps={pointerFocusProps} /> ) : ( @@ -466,7 +479,7 @@ export const ComposerPrimaryActions = memo(function ComposerPrimaryActions({ size="sm" className={cn("rounded-full", compact ? "h-9 px-3 sm:h-8" : "h-9 px-4 sm:h-8")} {...pointerFocusProps} - disabled={isSendBusy || isConnecting || isEnvironmentUnavailable} + disabled={sendUnavailable} > {isConnecting || isSendBusy ? "Sending..." : "Refine"} @@ -480,7 +493,7 @@ export const ComposerPrimaryActions = memo(function ComposerPrimaryActions({ size="sm" className="h-9 rounded-l-full rounded-r-none px-4 sm:h-8" {...pointerFocusProps} - disabled={isSendBusy || isConnecting || isEnvironmentUnavailable} + disabled={sendUnavailable} > {isConnecting || isSendBusy ? "Sending..." : "Implement"} @@ -493,17 +506,14 @@ export const ComposerPrimaryActions = memo(function ComposerPrimaryActions({ className="h-9 rounded-l-none rounded-r-full border-l-white/12 px-2 sm:h-8" aria-label="Implementation actions" {...pointerFocusProps} - disabled={isSendBusy || isConnecting || isEnvironmentUnavailable} + disabled={sendUnavailable} /> } > - void onImplementPlanInNewThread()} - > + void onImplementPlanInNewThread()}> Implement in a new thread {runtimeModeOptions.length > 1 ? ( @@ -560,19 +570,21 @@ export const ComposerPrimaryActions = memo(function ComposerPrimaryActions({ return ( ); }); + +const SEND_BUTTON_CLASS_NAME = + "flex h-9 w-9 enabled:cursor-pointer items-center justify-center rounded-full bg-primary text-primary-foreground transition-colors duration-150 hover:bg-primary-hover disabled:pointer-events-none disabled:opacity-30 sm:h-8 sm:w-8"; + +function SendArrowIcon() { + return ( + + ); +} + +/** + * The send button while nothing new can go (ComposerPrimaryActionsProps. + * sendBlockedReason): disabled, named by why, and saying it on hover. The + * hover lands on its wrapper, since a disabled button takes no pointer. + */ +function ComposerBlockedSendButton({ reason }: { reason: string }) { + return ( + + } + > + + + + {reason} + + + ); +} diff --git a/apps/web/src/components/chat/threadMoveNotice.tsx b/apps/web/src/components/chat/threadMoveNotice.tsx index 913880a2c..684399db4 100644 --- a/apps/web/src/components/chat/threadMoveNotice.tsx +++ b/apps/web/src/components/chat/threadMoveNotice.tsx @@ -9,11 +9,24 @@ * finish nor stop, and offers a way to keep working here instead: a new * thread forked from this one. * + * Before that, a move can wait for the thread to be free (Part 3b): its + * notice says when it goes, or what holds it, with Cancel (and Try again + * when held). One that ended without starting says why ("Didn't move: …") + * until the user dismisses it. + * * @module threadMoveNotice */ -import type { EnvironmentId, OrchestrationThreadMove, ThreadId } from "@threadlines/contracts"; +import { + type EnvironmentId, + type OrchestrationThreadMove, + type ThreadId, + ThreadMoveNotStartedPayload, + type ThreadMoveRequestId, + type ThreadPendingMove, +} from "@threadlines/contracts"; import { scopeThreadRef } from "@threadlines/client-runtime"; import { useNavigate } from "@tanstack/react-router"; +import * as Schema from "effect/Schema"; import { useCallback, useMemo, useState } from "react"; import { describeComputerLinkFailure } from "../../environments/links/computerLinks"; @@ -23,8 +36,14 @@ import { } from "../../environments/links/threadMoves"; import { usePrimaryEnvironmentId } from "../../environments/primary"; import { useSavedEnvironmentRuntimeStore } from "../../environments/runtime"; +import { useLocalStorage } from "../../hooks/useLocalStorage"; import { useStore } from "../../store"; -import { describeThreadMoveProgress, isThreadMoveStalled } from "../../threadMove"; +import { + describePendingMove, + describeThreadMoveProgress, + isThreadMoveStalled, + THREAD_MOVE_NOT_STARTED_ACTIVITY_KIND, +} from "../../threadMove"; import { buildThreadRouteParams } from "../../threadRoutes"; import type { Thread } from "../../types"; import { Button } from "../ui/button"; @@ -191,3 +210,183 @@ export function useThreadMoveNotice(input: { [canOpen, cancelling, move, onCancel, onContinueHere, onOpen, stalled], ); } + +export interface PendingMoveNoticeActions { + /** Cancels the move the notice shows (its own request id, never a newer one). */ + readonly onCancel: () => void; + readonly cancelling: boolean; + /** Asks the server to try a held move again now. */ + readonly onRetry: () => void; + readonly retrying: boolean; +} + +/** The notice for a move that waits for the thread to be free (Part 3b). */ +export function buildPendingMoveNotice(input: { + readonly pendingMove: ThreadPendingMove; + readonly actions: PendingMoveNoticeActions; +}): ComposerNotice { + const { pendingMove, actions } = input; + const words = describePendingMove(pendingMove); + return { + id: `thread-move-queued:${pendingMove.requestId}`, + severity: words.severity, + lead: words.lead, + wrapActions: words.retry, + actions: ( + <> + {words.retry ? ( + + ) : null} + + + ), + }; +} + +/** A request this app is sending for a thread's waiting move, and which. */ +interface PendingMoveCall { + readonly kind: "cancel" | "retry"; + readonly threadId: ThreadId; + readonly requestId: ThreadMoveRequestId; +} + +/** + * buildPendingMoveNotice for the open thread, with its actions: Cancel and + * Try again go to the thread's own computer, naming the request shown. + */ +export function usePendingMoveNotice(input: { + readonly thread: Pick | undefined; + readonly computerLabel: string; +}): ComposerNotice | null { + const { thread, computerLabel } = input; + const pendingMove = thread?.pendingMove ?? null; + const environmentId = thread?.environmentId ?? null; + const threadId = thread?.id ?? null; + const [call, setCall] = useState(null); + + const send = useCallback( + (kind: PendingMoveCall["kind"], requestId: ThreadMoveRequestId) => { + if (environmentId === null || threadId === null) return; + const current = { kind, threadId, requestId }; + setCall(current); + void (async () => { + try { + const client = requireThreadMovesClient(environmentId, computerLabel); + await (kind === "cancel" + ? client.unqueue({ threadId, requestId }) + : client.retry({ threadId, requestId })); + } catch (error) { + toastManager.add( + failureNotice({ + title: kind === "cancel" ? "Couldn't cancel the move" : "Couldn't try the move again", + reason: describeComputerLinkFailure(error, `${computerLabel} didn't answer.`), + }), + ); + } finally { + setCall((latest) => (latest === current ? null : latest)); + } + })(); + }, + [computerLabel, environmentId, threadId], + ); + + const requestId = pendingMove?.requestId ?? null; + const busyWith = (kind: PendingMoveCall["kind"]) => + call !== null && + call.kind === kind && + call.threadId === threadId && + call.requestId === requestId; + const cancelling = busyWith("cancel"); + const retrying = busyWith("retry"); + return useMemo( + () => + pendingMove === null + ? null + : buildPendingMoveNotice({ + pendingMove, + actions: { + onCancel: () => send("cancel", pendingMove.requestId), + cancelling, + onRetry: () => send("retry", pendingMove.requestId), + retrying, + }, + }), + [cancelling, pendingMove, retrying, send], + ); +} + +const isMoveNotStartedPayload = Schema.is(ThreadMoveNotStartedPayload); + +const MOVE_NOT_STARTED_DISMISSALS_STORAGE_KEY = "threadlines:move-not-started-dismissals:v1"; +const MoveNotStartedDismissalsSchema = Schema.Struct({ keys: Schema.Array(Schema.String) }); +/** Enough to outlast any notice still on screen; older keys fall off the front. */ +const MAX_MOVE_NOT_STARTED_DISMISSALS = 200; + +/** + * The thread's last queued move that ended without starting ("Didn't move: + * "), until dismissed on this device. Gone while another move waits or + * runs, which says more. + */ +export function useMoveNotStartedNotice(input: { + readonly thread: + | Pick + | undefined; +}): ComposerNotice | null { + const { thread } = input; + const [dismissals, setDismissals] = useLocalStorage( + MOVE_NOT_STARTED_DISMISSALS_STORAGE_KEY, + { keys: [] }, + MoveNotStartedDismissalsSchema, + ); + const busy = Boolean(thread?.move || thread?.pendingMove); + const activities = thread?.activities; + const latest = useMemo(() => { + if (!activities) return null; + for (let index = activities.length - 1; index >= 0; index -= 1) { + const activity = activities[index]; + if (activity?.kind !== THREAD_MOVE_NOT_STARTED_ACTIVITY_KIND) continue; + return isMoveNotStartedPayload(activity.payload) ? activity.payload : null; + } + return null; + }, [activities]); + const dismissalKey = + thread && latest ? `${thread.environmentId}:${thread.id}:${latest.requestId}` : null; + const dismissed = dismissalKey !== null && dismissals.keys.includes(dismissalKey); + const dismiss = useCallback(() => { + if (dismissalKey === null) return; + setDismissals((current) => + current.keys.includes(dismissalKey) + ? current + : { keys: [...current.keys, dismissalKey].slice(-MAX_MOVE_NOT_STARTED_DISMISSALS) }, + ); + }, [dismissalKey, setDismissals]); + return useMemo( + () => + latest === null || busy || dismissed + ? null + : { + id: `thread-move-not-started:${latest.requestId}`, + severity: "warning", + lead: "Didn't move:", + detail: latest.words, + dismissLabel: `Dismiss why it didn't move to ${latest.targetLabel}`, + onDismiss: dismiss, + }, + [busy, dismiss, dismissed, latest], + ); +} diff --git a/apps/web/src/components/sidebar/InboxRows.tsx b/apps/web/src/components/sidebar/InboxRows.tsx index 13d223749..ad132a305 100644 --- a/apps/web/src/components/sidebar/InboxRows.tsx +++ b/apps/web/src/components/sidebar/InboxRows.tsx @@ -77,7 +77,11 @@ import { import { Button } from "../ui/button"; import { useLinkRequestAnswer } from "./linkedComputers"; import { runtimeModeConfig } from "../../runtimeModeOptions"; -import { formatThreadMoveAccess, formatThreadMoveChanges } from "../../threadMove"; +import { + formatThreadMoveAccess, + formatThreadMoveChanges, + threadMoveStatusWord, +} from "../../threadMove"; import { ProjectFavicon } from "../ProjectFavicon"; import { ThreadHoverCard, @@ -450,14 +454,22 @@ export function ThreadProviderGlyph({ thread }: { thread: SidebarThreadSummary } * rows read it the same way, so a thread says the same thing wherever it sits. */ function resolveRowStatusParts(thread: SidebarThreadSummary, status: ThreadStatusPill | null) { - // Moving to a linked computer, or moved there: frozen here either way, so - // the one quiet word wins the slot over whatever it last did. - const moveWord = thread.moveStatus ?? null; - const statusWord = moveWord ?? inboxStatusWord(status); const isInFlight = status?.label === "Working" || status?.label === "Starting"; // In a room, an agent answering on the side while the others are idle: its // own name, and a clock from when it was asked. const isAnswering = status?.label === "Answering"; + const pillWord = inboxStatusWord(status); + // Moving to a linked computer, or moved there: frozen here either way, so + // the one quiet word wins the slot over whatever it last did. A move that + // still waits is quieter: what the thread is doing, or needs, comes first, + // and "move queued" shows once it is only waiting to go. + const moveWord = + thread.moveStatus === undefined || + (thread.moveStatus === "queued" && + (pillWord !== null || isInFlight || isAnswering || status?.label === "Waiting")) + ? null + : threadMoveStatusWord(thread.moveStatus); + const statusWord = moveWord ?? pillWord; // A follow-up starts its session before the new turn's row lands in the // projection, so for a beat `latestTurn` is still the previous, finished // turn -- its clock must not run under the new label, which is "Working" @@ -2345,7 +2357,7 @@ export const InboxDoneRow = memo(function InboxDoneRow(props: InboxDoneRowProps) : "text-muted-foreground/45", )} > - {thread.moveStatus ?? + {(thread.moveStatus && threadMoveStatusWord(thread.moveStatus)) ?? (checkBackAt !== null ? ( ) : doneAt ? ( diff --git a/apps/web/src/rpc/wsRpcClient.ts b/apps/web/src/rpc/wsRpcClient.ts index d436a0493..cc8af158e 100644 --- a/apps/web/src/rpc/wsRpcClient.ts +++ b/apps/web/src/rpc/wsRpcClient.ts @@ -534,6 +534,13 @@ export interface WsRpcClient { readonly preview: RpcUnaryMethod; readonly start: RpcUnaryMethod; readonly cancel: RpcUnaryMethod; + /** + * A move that waits for the thread to be free (Part 3b). Only servers + * whose preview can say `waitsFor` or an `unreachable` target serve these. + */ + readonly queue: RpcUnaryMethod; + readonly unqueue: RpcUnaryMethod; + readonly retry: RpcUnaryMethod; }; /** * Setting up a project on another computer (docs/design/fleet.md, Part 8), @@ -1409,6 +1416,9 @@ export function createWsRpcClient(transport: WsTransport): WsRpcClient { preview: (input) => transport.request((client) => client[WS_METHODS.movesPreview](input)), start: (input) => transport.request((client) => client[WS_METHODS.movesStart](input)), cancel: (input) => transport.request((client) => client[WS_METHODS.movesCancel](input)), + queue: (input) => transport.request((client) => client[WS_METHODS.movesQueue](input)), + unqueue: (input) => transport.request((client) => client[WS_METHODS.movesUnqueue](input)), + retry: (input) => transport.request((client) => client[WS_METHODS.movesRetry](input)), }, // Every call here is safe to send again: the reads are pure, a start is // the same job for the same setupId, and cancel and dismiss repeat as diff --git a/apps/web/src/session-logic.ts b/apps/web/src/session-logic.ts index 4a57ed264..61284b3ed 100644 --- a/apps/web/src/session-logic.ts +++ b/apps/web/src/session-logic.ts @@ -76,7 +76,11 @@ import { } from "./mcpAuthStatus"; import { filterSupersededManualContextCompactionActivities } from "./lib/contextCompactionActivities"; import { keyedMemo } from "./lib/derivationMemo"; -import { THREAD_MOVE_FAILED_ACTIVITY_KIND, THREAD_MOVED_IN_ACTIVITY_KIND } from "./threadMove"; +import { + THREAD_MOVE_FAILED_ACTIVITY_KIND, + THREAD_MOVE_NOT_STARTED_ACTIVITY_KIND, + THREAD_MOVED_IN_ACTIVITY_KIND, +} from "./threadMove"; import type { ChatMessage, @@ -3503,9 +3507,14 @@ function toDerivedWorkLogEntry( : null; const taskLabel = taskSummary || taskDetailAsLabel; const runtimeWarningDisplay = deriveRuntimeWarningDisplay(activity, payload); - // A dropped move says why under its "Couldn't move to …". + // A dropped move says why under its "Couldn't move to …", and a queued + // move that never started under its "Didn't move to …". const moveFailureReason = - activity.kind === THREAD_MOVE_FAILED_ACTIVITY_KIND ? asTrimmedString(payload?.reason) : null; + activity.kind === THREAD_MOVE_FAILED_ACTIVITY_KIND + ? asTrimmedString(payload?.reason) + : activity.kind === THREAD_MOVE_NOT_STARTED_ACTIVITY_KIND + ? asTrimmedString(payload?.words) + : null; const detail = moveFailureReason ? moveFailureReason : isTaskActivity diff --git a/apps/web/src/store.test.ts b/apps/web/src/store.test.ts index b9c0cea0d..922c11b32 100644 --- a/apps/web/src/store.test.ts +++ b/apps/web/src/store.test.ts @@ -19,6 +19,7 @@ import { ThreadId, ThreadParticipantId, ThreadMoveId, + ThreadMoveRequestId, TurnId, OrchestrationThread, type OrchestrationEvent, @@ -2362,6 +2363,120 @@ describe("incremental orchestration updates", () => { expect(summaryOf(state)?.moveStatus).toBe("moved"); }); + it("follows a move that waits for the thread to be free, and drops steps of an older one", () => { + const threadId = ThreadId.make("thread-1"); + const ref = scopeThreadRef(localEnvironmentId, threadId); + const requestId = ThreadMoveRequestId.make("queued-1"); + const macMini = { + environmentId: EnvironmentId.make("environment-mac-mini"), + label: "Mac mini", + }; + const agent = { instanceId: ProviderInstanceId.make("codex"), model: "gpt-5" }; + const apply = (state: AppState, event: OrchestrationEvent) => + applyOrchestrationEvent(state, event, localEnvironmentId); + const pendingOf = (state: AppState) => selectThreadByRef(state, ref)?.pendingMove; + const summaryOf = (state: AppState) => + localEnvironmentStateOf(state).sidebarThreadSummaryById[threadId]; + const queue = (id: typeof requestId) => + makeEvent("thread.move-queued", { + threadId, + pendingMove: { + requestId: id, + target: macMini, + agent, + mode: "when-done", + stage: "waiting", + requestedAt: "2026-02-27T00:00:01.000Z", + requestedBy: { kind: "user" }, + waitsFor: [{ code: "turn", words: "when this turn ends" }], + }, + }); + + let state = apply(makeState(makeThread({ id: threadId })), queue(requestId)); + expect(pendingOf(state)).toMatchObject({ requestId, stage: "waiting", target: macMini }); + expect(summaryOf(state)?.moveStatus).toBe("queued"); + + // The turn ended; Mac mini is offline: it waits, held, saying why. + state = apply( + state, + makeEvent("thread.move-queue-status-set", { + threadId, + requestId, + waitsFor: [], + hold: { kind: "target_unreachable", words: "Waiting for Mac mini to connect." }, + updatedAt: "2026-02-27T00:00:02.000Z", + }), + ); + // A step naming another request is not this one's. + state = apply( + state, + makeEvent("thread.move-quiescing", { + threadId, + requestId: ThreadMoveRequestId.make("queued-old"), + updatedAt: "2026-02-27T00:00:03.000Z", + }), + ); + expect(pendingOf(state)).toMatchObject({ + stage: "waiting", + waitsFor: [], + hold: { kind: "target_unreachable", words: "Waiting for Mac mini to connect." }, + }); + + state = apply( + state, + makeEvent("thread.move-quiescing", { + threadId, + requestId, + updatedAt: "2026-02-27T00:00:04.000Z", + }), + ); + expect(pendingOf(state)?.stage).toBe("quiescing"); + + // Cancelled: the thread is its own again. A late end of an old request changes nothing. + state = apply( + state, + makeEvent("thread.move-queue-ended", { + threadId, + requestId, + reason: "cancelled", + endedAt: "2026-02-27T00:00:05.000Z", + }), + ); + expect(pendingOf(state)).toBeUndefined(); + expect(summaryOf(state)?.moveStatus).toBeUndefined(); + + // Queued again, and started: the move takes its place in one step. + const secondRequestId = ThreadMoveRequestId.make("queued-2"); + state = apply(state, queue(secondRequestId)); + state = apply( + state, + makeEvent("thread.move-queue-ended", { + threadId, + requestId, + reason: "cancelled", + endedAt: "2026-02-27T00:00:06.000Z", + }), + ); + expect(pendingOf(state)?.requestId).toBe(secondRequestId); + state = apply( + state, + makeEvent("thread.move-started", { + threadId, + transferId: ThreadMoveId.make("move-from-queue"), + target: macMini, + agent, + startedAt: "2026-02-27T00:00:07.000Z", + queuedRequestId: secondRequestId, + }), + ); + expect(pendingOf(state)).toBeUndefined(); + expect(selectThreadByRef(state, ref)?.move).toMatchObject({ + phase: "exporting", + queuedRequestId: secondRequestId, + }); + expect(summaryOf(state)?.moveStatus).toBe("moving"); + }); + it("keeps a moved thread's imported history marked, and out of its latest turn", () => { const threadId = ThreadId.make("thread-1"); const ref = scopeThreadRef(localEnvironmentId, threadId); diff --git a/apps/web/src/store.ts b/apps/web/src/store.ts index 1030d8075..35f986ad2 100644 --- a/apps/web/src/store.ts +++ b/apps/web/src/store.ts @@ -40,6 +40,8 @@ import type { ScopedProjectRef, ScopedThreadRef, ThreadMovedFrom, + ThreadMoveRequestId, + ThreadPendingMove, ThreadRemoteOrigin, } from "@threadlines/contracts"; import { @@ -607,14 +609,42 @@ function remoteChildStartsEqual( */ function threadMoves(thread: { readonly move?: OrchestrationThreadMove | null | undefined; + readonly pendingMove?: ThreadPendingMove | null | undefined; readonly movedFrom?: ThreadMovedFrom | null | undefined; -}): Pick { +}): Pick { return { ...(thread.move ? { move: thread.move } : {}), + ...(thread.pendingMove ? { pendingMove: thread.pendingMove } : {}), ...(thread.movedFrom ? { movedFrom: thread.movedFrom } : {}), }; } +/** Two copies of a thread's waiting move, as the notice and the row read them. */ +function pendingMovesEqual( + left: ThreadPendingMove | null | undefined, + right: ThreadPendingMove | null | undefined, +): boolean { + const a = left ?? null; + const b = right ?? null; + if (a === null || b === null) return a === b; + const aWaits = a.waitsFor ?? []; + const bWaits = b.waitsFor ?? []; + return ( + a.requestId === b.requestId && + a.target.environmentId === b.target.environmentId && + a.target.label === b.target.label && + a.mode === b.mode && + a.stage === b.stage && + a.requestedBy.kind === b.requestedBy.kind && + (a.hold?.kind ?? null) === (b.hold?.kind ?? null) && + (a.hold?.words ?? null) === (b.hold?.words ?? null) && + aWaits.length === bWaits.length && + aWaits.every( + (wait, index) => wait.code === bWaits[index]?.code && wait.words === bWaits[index]?.words, + ) + ); +} + function threadMovesEqual( left: OrchestrationThreadMove | null | undefined, right: OrchestrationThreadMove | null | undefined, @@ -633,6 +663,7 @@ function threadMovesEqual( a.totalBytes === b.totalBytes && a.cancelRequested === b.cancelRequested && a.startedAt === b.startedAt && + a.queuedRequestId === b.queuedRequestId && a.movedAway?.threadId === b.movedAway?.threadId && a.movedAway?.movedAt === b.movedAway?.movedAt; } @@ -671,6 +702,7 @@ function sidebarLineage( | "remoteChildStarts" | "localChildStarts" | "move" + | "pendingMove" >, ): Pick< SidebarThreadSummary, @@ -691,7 +723,7 @@ function sidebarLineage( | "localChildStarts" | "moveStatus" > { - const moveStatus = threadMoveStatus(thread.move); + const moveStatus = threadMoveStatus(thread); return { scheduledRun: thread.scheduledRun ?? null, worktreeOrigin: thread.worktreeOrigin ?? null, @@ -1440,6 +1472,7 @@ function threadShellsEqual(left: ThreadShell | undefined, right: ThreadShell): b remoteChildStartsEqual(left.remoteChildStarts, right.remoteChildStarts) && localChildStartsEqual(left.localChildStarts, right.localChildStarts) && threadMovesEqual(left.move, right.move) && + pendingMovesEqual(left.pendingMove, right.pendingMove) && movedFromEqual(left.movedFrom, right.movedFrom) && doneOverridesEqual(left.doneOverride, right.doneOverride) && left.lastSeenAt === right.lastSeenAt && @@ -2316,6 +2349,26 @@ function updateThreadMove( }); } +/** + * Apply a step of one queued move (Part 3b) to a thread's `pendingMove`: only + * while that request is the thread's own, so a late step of a move that was + * cancelled or replaced changes nothing. The change returns null to clear it. + */ +function updatePendingMove( + state: EnvironmentState, + threadId: ThreadId, + requestId: ThreadMoveRequestId, + change: (current: ThreadPendingMove) => ThreadPendingMove | null, +): EnvironmentState { + return updateThreadState(state, threadId, (thread) => { + if (!thread.pendingMove || thread.pendingMove.requestId !== requestId) return thread; + const next = change(thread.pendingMove); + if (next !== null) return { ...thread, pendingMove: next }; + const { pendingMove: _ended, ...rest } = thread; + return rest; + }); +} + function buildProjectState( projects: ReadonlyArray, ): Pick { @@ -3619,18 +3672,59 @@ function applyEnvironmentOrchestrationEvent( // names its move, so a late one from a move that already ended is // dropped. Like a done mark, none moves `updatedAt`. case "thread.move-started": + return updateThreadState(state, event.payload.threadId, (thread) => { + const queuedRequestId = event.payload.queuedRequestId; + // The queued move it started from ends in the same decision + // (`thread.move-queue-ended`); a thread never shows both. + const { pendingMove: _started, ...rest } = thread; + return { + ...(thread.pendingMove && thread.pendingMove.requestId !== queuedRequestId + ? thread + : rest), + move: { + transferId: event.payload.transferId, + target: event.payload.target, + agent: event.payload.agent, + phase: "exporting", + cancelRequested: false, + startedAt: event.payload.startedAt, + ...(queuedRequestId !== undefined ? { queuedRequestId } : {}), + }, + }; + }); + + // A move that waits for the thread to be free (Part 3b). Each step names + // its request, so a late one from a cancelled or replaced move is dropped. + case "thread.move-queued": return updateThreadState(state, event.payload.threadId, (thread) => ({ ...thread, - move: { - transferId: event.payload.transferId, - target: event.payload.target, - agent: event.payload.agent, - phase: "exporting", - cancelRequested: false, - startedAt: event.payload.startedAt, - }, + pendingMove: event.payload.pendingMove, })); + case "thread.move-quiescing": + return updatePendingMove( + state, + event.payload.threadId, + event.payload.requestId, + (pendingMove) => ({ ...pendingMove, stage: "quiescing" }), + ); + + case "thread.move-queue-status-set": + return updatePendingMove( + state, + event.payload.threadId, + event.payload.requestId, + (pendingMove) => ({ + ...pendingMove, + waitsFor: event.payload.waitsFor, + hold: event.payload.hold, + }), + ); + + // Its "Didn't move" activity, when it didn't start, arrives on its own. + case "thread.move-queue-ended": + return updatePendingMove(state, event.payload.threadId, event.payload.requestId, () => null); + case "thread.move-progressed": return updateThreadMove(state, event.payload.threadId, event.payload.transferId, (move) => { const { sentBytes: _sent, totalBytes: _total, ...rest } = move; diff --git a/apps/web/src/threadMove.test.ts b/apps/web/src/threadMove.test.ts index 3e0039dba..94c22797c 100644 --- a/apps/web/src/threadMove.test.ts +++ b/apps/web/src/threadMove.test.ts @@ -6,11 +6,15 @@ import { ProviderInstanceId, ThreadId, ThreadMoveId, + ThreadMoveRequestId, type ThreadMoveTargetPreview, + type ThreadPendingMove, + TurnId, } from "@threadlines/contracts"; import { describe, expect, it } from "vite-plus/test"; import { + describePendingMove, describeThreadMoveProgress, formatThreadMoveChanges, isThreadMoveOffered, @@ -121,6 +125,42 @@ describe("describeThreadMoveProgress", () => { }); }); +describe("describePendingMove", () => { + const pending = (overrides: Partial = {}): ThreadPendingMove => ({ + requestId: ThreadMoveRequestId.make("queued-1"), + target: { environmentId: MAC_MINI, label: "Mac mini" }, + agent: { instanceId: ProviderInstanceId.make("codex"), model: "gpt-5" }, + mode: "when-done", + stage: "waiting", + requestedAt: "2026-10-22T09:00:00.000Z", + requestedBy: { kind: "user" }, + waitsFor: [{ code: "approval", words: "once you answer" }], + ...overrides, + }); + + it("says when it goes, in the agent's name when the agent asked, and what holds it", () => { + expect(describePendingMove(pending()).lead).toBe("Moves to Mac mini once you answer."); + expect( + describePendingMove( + pending({ + requestedBy: { kind: "agent", turnId: TurnId.make("turn-1") }, + waitsFor: [{ code: "turn", words: "when this turn ends" }], + }), + ).lead, + ).toBe("Your agent asked to move this thread to Mac mini when this turn ends."); + expect(describePendingMove(pending({ stage: "quiescing", waitsFor: [] })).lead).toBe( + "Stopping the agent to move to Mac mini…", + ); + expect( + describePendingMove( + pending({ + hold: { kind: "target_unreachable", words: "Waiting for Mac mini to connect." }, + }), + ), + ).toEqual({ lead: "Waiting for Mac mini to connect.", severity: "info", retry: true }); + }); +}); + describe("isThreadMoveStalled", () => { it("holds an unfinished move whose link is gone, revoked or can't reach the other computer", () => { expect(isThreadMoveStalled({ move: move(), outbound: [link()] })).toBe(false); @@ -181,6 +221,16 @@ describe("picking where and with whom", () => { }, }); expect(pickDefaultMoveTarget([refused, suggested])).toBe(suggested); + // One out of reach comes before one that can't take it, after one that can. + const offline = target({ + environmentId: EnvironmentId.make("environment-laptop"), + label: "Laptop", + refusal: "Laptop can't be reached right now.", + refusalCode: "unreachable", + agents: [], + }); + expect(pickDefaultMoveTarget([refused, offline])).toBe(offline); + expect(pickDefaultMoveTarget([offline, suggested])).toBe(suggested); expect(pickDefaultMoveAgent(suggested)).toEqual({ instanceId: "claudeAgent", model: "claude-opus-5-5", diff --git a/apps/web/src/threadMove.ts b/apps/web/src/threadMove.ts index 2940b7aec..3aac6082f 100644 --- a/apps/web/src/threadMove.ts +++ b/apps/web/src/threadMove.ts @@ -6,12 +6,16 @@ * * @module threadMove */ -import type { - ModelSelection, - OrchestrationThreadMove, - OutboundComputerLink, - ThreadMoveChanges, - ThreadMoveTargetPreview, +import { + type ModelSelection, + type OrchestrationThreadMove, + type OutboundComputerLink, + type ThreadMoveChanges, + ThreadMoveNotStartedActivityKind, + type ThreadMoveTargetPreview, + type ThreadMoveWait, + type ThreadMoveWaitCode, + type ThreadPendingMove, } from "@threadlines/contracts"; import type { ThreadMoveStatus } from "./types"; @@ -26,13 +30,28 @@ export const THREAD_MOVED_IN_ACTIVITY_KIND = "thread.moved-in"; * payload `{ targetLabel, reason }`. */ export const THREAD_MOVE_FAILED_ACTIVITY_KIND = "thread.move-failed"; +/** + * A queued move that ended without starting (Part 3b): "Didn't move to …", + * payload a ThreadMoveNotStartedPayload (`{ requestId, targetLabel, reason, words }`). + */ +export const THREAD_MOVE_NOT_STARTED_ACTIVITY_KIND = ThreadMoveNotStartedActivityKind; + +/** + * "queued" while a move waits for the thread to be free (Part 3b), "moving" + * until the other computer took it, "moved" after; undefined for none. + */ +export function threadMoveStatus(thread: { + readonly move?: OrchestrationThreadMove | null | undefined; + readonly pendingMove?: ThreadPendingMove | null | undefined; +}): ThreadMoveStatus | undefined { + const move = thread.move ?? null; + if (move !== null) return move.movedAway ? "moved" : "moving"; + return thread.pendingMove ? "queued" : undefined; +} -/** "moving" until the other computer took it, "moved" after; undefined for neither. */ -export function threadMoveStatus( - move: OrchestrationThreadMove | null | undefined, -): ThreadMoveStatus | undefined { - if (!move) return undefined; - return move.movedAway ? "moved" : "moving"; +/** A move status as a sidebar row says it: "move queued", "moving", "moved". */ +export function threadMoveStatusWord(status: ThreadMoveStatus): string { + return status === "queued" ? "move queued" : status; } /** @@ -137,11 +156,27 @@ export function isThreadMoveStalled(input: { return link === undefined || link.state !== "active" || link.connection === "offline"; } -/** The computer to offer first: the first that can take the thread, else the first. */ +/** + * A computer a move can be asked of: one that can take the thread, or one + * that can't be reached right now, which a queued move waits for. + */ +export function isThreadMoveTargetPickable(target: ThreadMoveTargetPreview): boolean { + return target.refusal === undefined || target.refusalCode === "unreachable"; +} + +/** + * The computer to offer first: the first that can take the thread, else the + * first a queued move can wait for, else the first. + */ export function pickDefaultMoveTarget( targets: ReadonlyArray, ): ThreadMoveTargetPreview | null { - return targets.find((target) => target.refusal === undefined) ?? targets[0] ?? null; + return ( + targets.find((target) => target.refusal === undefined) ?? + targets.find(isThreadMoveTargetPickable) ?? + targets[0] ?? + null + ); } /** One of a target's agents and models, as the agent picker holds it. */ @@ -191,3 +226,84 @@ export function threadMoveAgentSelection( } return { instanceId: choice.instanceId as ModelSelection["instanceId"], model: choice.model }; } + +// --------------------------------------------------------------------------- +// A move that waits for the thread to be free (docs/design/fleet.md, Part 3b) +// --------------------------------------------------------------------------- + +/** What a busy thread is doing, as it reads after "This thread is busy: ". */ +const WAIT_DOING: Record = { + turn: "its agent is working on a turn", + approval: "it's waiting for your approval", + question: "it's waiting for your answer", + side_turn: "a side answer is running", + background: "background work is still running", + command_run: "a command is running", + workflow_run: "a workflow is running", + voice: "a voice chat is on", + shared_checkout: "another thread is working in the same folder", +}; + +/** What a busy thread is doing, in one line: "This thread is busy: its agent is working on a turn." */ +export function describeThreadMoveWaits(waits: ReadonlyArray): string { + const doing = [...new Set(waits.map((wait) => WAIT_DOING[wait.code]))]; + const last = doing.at(-1); + if (last === undefined) return "This thread is free."; + const list = doing.length === 1 ? last : `${doing.slice(0, -1).join(", ")} and ${last}`; + return `This thread is busy: ${list}.`; +} + +/** When a move that waits starts: "Moves to Mac mini when this turn ends." */ +export function describeThreadMoveWhenDone( + label: string, + waits: ReadonlyArray, +): string { + const first = waits[0]; + return first ? `Moves to ${label} ${first.words}.` : `Moves to ${label} as soon as it's free.`; +} + +/** Why a thread whose move waits takes no new message: the send button's words. */ +export function pendingMoveSendBlockReason(pendingMove: ThreadPendingMove): string { + return `This thread is waiting to move to ${pendingMove.target.label}. Cancel the move to keep working here.`; +} + +/** The composer notice's words for a move that waits. */ +export interface PendingMoveNoticeWords { + readonly lead: string; + readonly severity: "info" | "warning"; + /** Held (the target is offline, or a stop wasn't confirmed): Try again asks the server to look now. */ + readonly retry: boolean; +} + +/** + * What a waiting move's notice says: what holds it, when something does; + * else when it goes ("Moves to Mac mini when this turn ends."), in the + * agent's name when the agent asked for it. + */ +export function describePendingMove(pendingMove: ThreadPendingMove): PendingMoveNoticeWords { + const label = pendingMove.target.label; + const hold = pendingMove.hold ?? null; + if (hold !== null) { + return { + lead: hold.words, + severity: hold.kind === "stop_unconfirmed" ? "warning" : "info", + retry: true, + }; + } + if (pendingMove.stage === "quiescing") { + return { lead: `Stopping the agent to move to ${label}…`, severity: "info", retry: false }; + } + const words = pendingMove.waitsFor?.[0]?.words ?? null; + if (pendingMove.requestedBy.kind === "agent") { + return { + lead: `Your agent asked to move this thread to ${label} ${words ?? "when this turn ends"}.`, + severity: "info", + retry: false, + }; + } + return { + lead: words === null ? `Getting ready to move to ${label}…` : `Moves to ${label} ${words}.`, + severity: "info", + retry: false, + }; +} diff --git a/apps/web/src/types.ts b/apps/web/src/types.ts index 8f1d41c28..faa604077 100644 --- a/apps/web/src/types.ts +++ b/apps/web/src/types.ts @@ -37,6 +37,7 @@ import type { ThreadRemoteOrigin, OrchestrationThreadMove, ThreadMovedFrom, + ThreadPendingMove, ProjectKind, ProjectScript as ContractProjectScript, ThreadId, @@ -283,6 +284,8 @@ export interface Thread { remoteOrigin?: ThreadRemoteOrigin | null; /** See ThreadShell.move. */ move?: OrchestrationThreadMove | null; + /** See ThreadShell.pendingMove. */ + pendingMove?: ThreadPendingMove | null; /** See ThreadShell.movedFrom. */ movedFrom?: ThreadMovedFrom | null; /** See ThreadShell.doneOverride. */ @@ -417,6 +420,12 @@ export interface ThreadShell { * null: neither. */ move?: OrchestrationThreadMove | null; + /** + * A move waiting for the thread to be free (ThreadPendingMove, Part 3b): + * the thread takes no new work until it starts or is cancelled. Absent or + * null: none. A thread has this or `move`, never both. + */ + pendingMove?: ThreadPendingMove | null; /** Moved here from another computer (ThreadMovedFrom). Absent or null: started here. */ movedFrom?: ThreadMovedFrom | null; /** @@ -446,8 +455,11 @@ export interface ThreadTurnState { pendingSourceProposedPlan?: OrchestrationLatestTurn["sourceProposedPlan"]; } -/** A thread moving to a linked computer, or moved there (ThreadShell.move). */ -export type ThreadMoveStatus = "moving" | "moved"; +/** + * A thread whose move waits for it to be free (ThreadShell.pendingMove), or + * moving to a linked computer, or moved there (ThreadShell.move). + */ +export type ThreadMoveStatus = "queued" | "moving" | "moved"; export interface SidebarThreadSummary { id: ThreadId; @@ -534,7 +546,7 @@ export interface SidebarThreadSummary { localChildStarts?: ReadonlyArray; /** Messages queued behind the running turn; a child with any is not finished. */ queuedFollowUpCount?: number; - /** See ThreadShell.move: only how far it got, for the row's status word. Absent: neither. */ + /** See ThreadShell.move and .pendingMove: only how far it got, for the row's status word. Absent: none. */ moveStatus?: ThreadMoveStatus; /** See Thread.checkBacks. */ checkBacks?: ReadonlyArray; From fe3529937ce9a6dcc68728d11c66b95bfbb396a3 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:46:21 -0500 Subject: [PATCH 32/48] fix(web): a waiting move's notice gives its words the full row on a phone --- apps/web/src/components/chat/threadMoveNotice.tsx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/apps/web/src/components/chat/threadMoveNotice.tsx b/apps/web/src/components/chat/threadMoveNotice.tsx index 684399db4..1ba1dab20 100644 --- a/apps/web/src/components/chat/threadMoveNotice.tsx +++ b/apps/web/src/components/chat/threadMoveNotice.tsx @@ -231,7 +231,8 @@ export function buildPendingMoveNotice(input: { id: `thread-move-queued:${pendingMove.requestId}`, severity: words.severity, lead: words.lead, - wrapActions: words.retry, + // On a phone the words get the row's whole width; they run long. + wrapActions: true, actions: ( <> {words.retry ? ( From 566293021a6213b3489a9fbcd6973774c5e7cce1 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:53:00 -0500 Subject: [PATCH 33/48] feat(web): every move goes through the queue where the server queues moves The dialog reads the server's threadMoveQueue capability instead of guessing from the preview: with it, free threads queue too; without it, moves.start as before. A move to a computer out of reach sends no agent, and the dialog says it uses the agent that computer suggests. --- .../ChatView.move-thread.browser.tsx | 90 ++++++++++++++++++- .../components/ThreadMoveDialogContent.tsx | 59 ++++++------ .../web/src/environments/links/threadMoves.ts | 27 ++++-- 3 files changed, 135 insertions(+), 41 deletions(-) diff --git a/apps/web/src/components/ChatView.move-thread.browser.tsx b/apps/web/src/components/ChatView.move-thread.browser.tsx index 9bfd573ed..2c8572512 100644 --- a/apps/web/src/components/ChatView.move-thread.browser.tsx +++ b/apps/web/src/components/ChatView.move-thread.browser.tsx @@ -83,8 +83,20 @@ vi.mock("../lib/gitStatusState", () => ({ /** This computer's server takes links and moves threads to linked computers. */ function withThreadMoveFixture(nextFixture: TestFixture): void { + withMoveCapabilities(nextFixture, { threadMove: true }); +} + +/** Like withThreadMoveFixture, on a server that also queues moves (Part 3b). */ +function withThreadMoveQueueFixture(nextFixture: TestFixture): void { + withMoveCapabilities(nextFixture, { threadMove: true, threadMoveQueue: true }); +} + +function withMoveCapabilities( + nextFixture: TestFixture, + moves: { readonly threadMove: true; readonly threadMoveQueue?: true }, +): void { withComputerLinksFixture(nextFixture); - const capabilities = { ...nextFixture.serverConfig.environment.capabilities, threadMove: true }; + const capabilities = { ...nextFixture.serverConfig.environment.capabilities, ...moves }; nextFixture.serverConfig = { ...nextFixture.serverConfig, environment: { ...nextFixture.serverConfig.environment, capabilities }, @@ -447,7 +459,7 @@ describe("ChatView (full app): moving a thread to another computer", () => { targetMessageId: "msg-user-move-busy" as MessageId, targetText: "move busy", }), - configureFixture: withThreadMoveFixture, + configureFixture: withThreadMoveQueueFixture, resolveRpc: (body) => { if (body._tag === WS_METHODS.movesPreview) { return { @@ -514,6 +526,80 @@ describe("ChatView (full app): moving a thread to another computer", () => { } }); + it("queues a free thread's move too, to a computer out of reach with the agent it suggests", async () => { + setPrimaryLinksSnapshot( + fleetLinksSnapshot({ + outbound: [fleetOutboundLink(FLEET_OFFLINE_ENVIRONMENT_ID, "Mac mini")], + }), + ); + const queues: Array = []; + const mounted = await mountChatView({ + viewport: WIDE_FOOTER_VIEWPORT, + snapshot: createSnapshotForTargetUser({ + targetMessageId: "msg-user-move-offline" as MessageId, + targetText: "move offline", + }), + configureFixture: withThreadMoveQueueFixture, + resolveRpc: (body) => { + if (body._tag === WS_METHODS.movesPreview) { + return { + changes: FLEET_MOVE_PREVIEW.changes, + targets: [ + { + environmentId: FLEET_OFFLINE_ENVIRONMENT_ID, + label: "Mac mini", + refusal: "Mac mini can't be reached right now.", + refusalCode: "unreachable", + agents: [], + }, + ], + } satisfies ThreadMovePreview; + } + if (body._tag === WS_METHODS.movesQueue) { + queues.push(body); + return { requestId: QUEUED_REQUEST_ID }; + } + return undefined; + }, + }); + + try { + await waitForServerConfigToApply(); + await saveMacMini("disconnected"); + await vi.waitFor(() => { + expect(readThreadMoveOffered(LOCAL_ENVIRONMENT_ID, undefined)).toBe(true); + }); + await openCommandPaletteFromTrigger(); + await page + .getByTestId("command-palette") + .getByText("Move to another computer", { exact: true }) + .click(); + const dialog = page.getByTestId("thread-move-dialog"); + await expect + .element(dialog.getByTestId("thread-move-waits-to-connect")) + .toHaveTextContent( + "Mac mini can't be reached right now. The move waits for it to connect, then uses the agent Mac mini suggests.", + ); + await expect + .element(dialog.getByTestId("thread-move-confirm")) + .toHaveTextContent("Move when it connects"); + await dialog.getByTestId("thread-move-confirm").click(); + await expect.element(page.getByTestId("thread-move-dialog")).not.toBeInTheDocument(); + expect(queues).toHaveLength(1); + expect(queues[0]).toEqual( + expect.objectContaining({ + threadId: THREAD_ID, + targetEnvironmentId: FLEET_OFFLINE_ENVIRONMENT_ID, + mode: "when-done", + }), + ); + // No agent: Mac mini's suggestion is picked once the move starts. + expect(queues[0]).not.toHaveProperty("agent"); + } finally { + await mounted.cleanup(); + } + }); + it("says when a waiting move goes, holds new messages, and cancels the move it shows", async () => { setPrimaryLinksSnapshot( fleetLinksSnapshot({ diff --git a/apps/web/src/components/ThreadMoveDialogContent.tsx b/apps/web/src/components/ThreadMoveDialogContent.tsx index bcf50d35f..a8c107f90 100644 --- a/apps/web/src/components/ThreadMoveDialogContent.tsx +++ b/apps/web/src/components/ThreadMoveDialogContent.tsx @@ -17,7 +17,10 @@ import { useNavigate } from "@tanstack/react-router"; import { useEffect, useMemo, useRef, useState } from "react"; import { describeComputerLinkFailure } from "../environments/links/computerLinks"; -import { requireThreadMovesClient } from "../environments/links/threadMoves"; +import { + requireThreadMovesClient, + useThreadMoveQueueSupported, +} from "../environments/links/threadMoves"; import { useProjectSetupComputerReady, useProjectSetupTargets, @@ -71,13 +74,9 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread const projectId = useStore( (state) => selectSidebarThreadSummaryByRef(state, threadRef)?.projectId ?? null, ); - // The agent a move to a computer that can't be reached asks for: the - // thread's own, since that computer can't say which it has. - const threadModelSelection = useStore( - (state) => - state.environmentStateById[threadRef.environmentId]?.threadShellById[threadRef.threadId] - ?.modelSelection ?? null, - ); + // The thread's computer queues moves (Part 3b): every move goes through + // the queue, and a free thread's starts at once. Older servers only start. + const queues = useThreadMoveQueueSupported(threadRef.environmentId); const computerLabel = useComputerLabel()(threadRef.environmentId, threadRef.environmentId); const [previewState, setPreviewState] = useState({ status: "loading" }); const [targetId, setTargetId] = useState(null); @@ -133,44 +132,36 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread const preview = previewState.status === "ready" ? previewState.preview : null; const target = preview?.targets.find((candidate) => candidate.environmentId === targetId) ?? null; // Busy: the move can wait for the thread to be free, or stop it first. - const waits = preview?.waitsFor ?? []; + const waits = queues ? (preview?.waitsFor ?? []) : []; const busy = waits.length > 0; - // Out of reach: a move to it waits for it to connect. - const unreachable = target?.refusalCode === "unreachable"; - // Only a server that queues moves says either, so asking it to queue is - // safe; anything else starts the move at once, as every server does. - const queues = busy || unreachable; + // Out of reach: a queued move to it waits for it to connect, and runs + // with the agent it suggests then (no agent sent). + const unreachable = queues && target?.refusalCode === "unreachable"; const agentSelection: ModelSelection | null = - target === null - ? null - : unreachable - ? threadModelSelection - : agent !== null - ? threadMoveAgentSelection(target, agent) - : null; + target !== null && agent !== null ? threadMoveAgentSelection(target, agent) : null; const canMove = preview !== null && preview.refusal === undefined && target !== null && - isThreadMoveTargetPickable(target) && - agentSelection !== null && + (target.refusal === undefined || unreachable) && + (agentSelection !== null || unreachable) && !starting; const start = async () => { - if (!canMove || target === null || agentSelection === null) return; + if (!canMove || target === null) return; setStarting(true); setStartError(null); try { const client = requireThreadMovesClient(threadRef.environmentId, computerLabel); - const input = { - threadId: threadRef.threadId, - targetEnvironmentId: target.environmentId, - agent: agentSelection, - }; + const input = { threadId: threadRef.threadId, targetEnvironmentId: target.environmentId }; if (queues) { - await client.queue({ ...input, mode: busy ? mode : "when-done" }); - } else { - await client.start(input); + await client.queue({ + ...input, + ...(unreachable || agentSelection === null ? {} : { agent: agentSelection }), + mode: busy ? mode : "when-done", + }); + } else if (agentSelection !== null) { + await client.start({ ...input, agent: agentSelection }); } closeThreadMoveDialog(); } catch (error) { @@ -283,8 +274,8 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread className="text-[12.5px] leading-[18px] text-muted-foreground" data-testid="thread-move-waits-to-connect" > - {target.refusal} The move waits for it to connect, and keeps the agent this - thread uses here. + {target.refusal} The move waits for it to connect, then uses the agent{" "} + {target.label} suggests.

) : ( environmentId === null ? null : (state.byId[environmentId]?.descriptor ?? null), ); + if (environmentId === null) return null; + return primary?.environmentId === environmentId ? primary : saved; +} + +/** + * Whether the computer `environmentId` queues moves (`moves.queue`, Part 3b), + * kept current. With it every move goes through the queue, a free thread's + * included; without it, only `moves.start` exists there. + */ +export function useThreadMoveQueueSupported(environmentId: EnvironmentId | null): boolean { + return useEnvironmentDescriptor(environmentId)?.capabilities.threadMoveQueue === true; +} + +/** isThreadMoveOffered for a thread on `environmentId`, kept current. */ +export function useThreadMoveOffered( + environmentId: EnvironmentId | null, + moveStatus: ThreadMoveStatus | undefined, +): boolean { + const descriptor = useEnvironmentDescriptor(environmentId); const outbound = useComputerLinksStore((state) => environmentId === null ? null : liveOutbound(state.byEnvironmentId[environmentId]), ); if (environmentId === null) return false; - const descriptor = primary?.environmentId === environmentId ? primary : saved; return isThreadMoveOffered({ supportsThreadMove: supportsThreadMove(descriptor), outbound, From ed06cca5db0f6a5d7c92f8c9916acb519be0dc7d Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 23:25:22 -0500 Subject: [PATCH 34/48] fix(web): the move dialog says what the agent is doing, not that the thread is busy The lead reads from the wait codes in at most two short sentences ("The agent is still working on its reply."), and the choices read "Move when the agent finishes" and "Stop the agent and move now". --- .../ChatView.move-thread.browser.tsx | 25 +++--- apps/web/src/components/ThreadMoveDialog.tsx | 5 +- .../components/ThreadMoveDialogContent.tsx | 28 +++---- .../src/components/chat/threadMoveNotice.tsx | 12 +-- apps/web/src/threadMove.ts | 77 ++++++++++++------- 5 files changed, 90 insertions(+), 57 deletions(-) diff --git a/apps/web/src/components/ChatView.move-thread.browser.tsx b/apps/web/src/components/ChatView.move-thread.browser.tsx index 2c8572512..8bd9ae1a5 100644 --- a/apps/web/src/components/ChatView.move-thread.browser.tsx +++ b/apps/web/src/components/ChatView.move-thread.browser.tsx @@ -445,7 +445,7 @@ describe("ChatView (full app): moving a thread to another computer", () => { } }); - it("offers a busy thread two ways to move, and queues the one picked", async () => { + it("offers a thread whose agent is still working two ways to move, and queues the one picked", async () => { setPrimaryLinksSnapshot( fleetLinksSnapshot({ outbound: [fleetOutboundLink(FLEET_OFFLINE_ENVIRONMENT_ID, "Mac mini")], @@ -456,8 +456,8 @@ describe("ChatView (full app): moving a thread to another computer", () => { const mounted = await mountChatView({ viewport: WIDE_FOOTER_VIEWPORT, snapshot: createSnapshotForTargetUser({ - targetMessageId: "msg-user-move-busy" as MessageId, - targetText: "move busy", + targetMessageId: "msg-user-move-working" as MessageId, + targetText: "move working", }), configureFixture: withThreadMoveQueueFixture, resolveRpc: (body) => { @@ -491,21 +491,28 @@ describe("ChatView (full app): moving a thread to another computer", () => { .getByText("Move to another computer", { exact: true }) .click(); const dialog = page.getByTestId("thread-move-dialog"); - // What it's busy with, then the two ways: when it's done comes picked. + // What the agent is doing, then the two ways: when it finishes comes picked. await expect - .element(dialog.getByTestId("thread-move-busy")) + .element(dialog.getByTestId("thread-move-doing")) .toHaveTextContent( - "This thread is busy: its agent is working on a turn and background work is still running.", + "The agent is still working on its reply. Something it started is still running too.", ); await expect .element(dialog.getByTestId("thread-move-mode-when-done")) - .toMatchTextContent("Move when it's doneMoves to Mac mini when this turn ends."); + .toMatchTextContent( + "Move when the agent finishesWaits for the agent to finish, then moves to Mac mini.", + ); + await expect + .element(dialog.getByTestId("thread-move-mode-stop-now")) + .toMatchTextContent( + "Stop the agent and move nowStops the agent and anything it started, then moves.", + ); await expect - .element(dialog.getByRole("radio", { name: /Move when it's done/u })) + .element(dialog.getByRole("radio", { name: /Move when the agent finishes/u })) .toBeChecked(); await expect .element(dialog.getByTestId("thread-move-confirm")) - .toHaveTextContent("Move when done"); + .toHaveTextContent("Move when it finishes"); await dialog.getByTestId("thread-move-mode-stop-now").click(); await expect diff --git a/apps/web/src/components/ThreadMoveDialog.tsx b/apps/web/src/components/ThreadMoveDialog.tsx index 63094e56d..700d095c7 100644 --- a/apps/web/src/components/ThreadMoveDialog.tsx +++ b/apps/web/src/components/ThreadMoveDialog.tsx @@ -3,8 +3,9 @@ * every entry point (the sidebar's thread menu, the command palette, the chat * header). Each asks through openThreadMoveDialog; the host mounted once at * the root shows it. The dialog asks the thread's computer where the thread - * could go and what would go with it, then starts the move there, or, for a - * busy thread or a computer out of reach, queues one that waits (Part 3b). + * could go and what would go with it, then queues the move there (Part 3b: + * it starts at once, or once the agent finishes, or once the other computer + * connects), or starts it, on a computer too old to queue moves. */ import type { ScopedThreadRef } from "@threadlines/contracts"; import { Suspense } from "react"; diff --git a/apps/web/src/components/ThreadMoveDialogContent.tsx b/apps/web/src/components/ThreadMoveDialogContent.tsx index a8c107f90..37f7cfd71 100644 --- a/apps/web/src/components/ThreadMoveDialogContent.tsx +++ b/apps/web/src/components/ThreadMoveDialogContent.tsx @@ -28,7 +28,6 @@ import { import { selectSidebarThreadSummaryByRef, useStore } from "../store"; import { describeThreadMoveWaits, - describeThreadMoveWhenDone, formatThreadMoveChanges, isThreadMoveTargetPickable, pickDefaultMoveAgent, @@ -131,9 +130,9 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread const preview = previewState.status === "ready" ? previewState.preview : null; const target = preview?.targets.find((candidate) => candidate.environmentId === targetId) ?? null; - // Busy: the move can wait for the thread to be free, or stop it first. + // Still working: the move can wait for the agent to finish, or stop it first. const waits = queues ? (preview?.waitsFor ?? []) : []; - const busy = waits.length > 0; + const working = waits.length > 0; // Out of reach: a queued move to it waits for it to connect, and runs // with the agent it suggests then (no agent sent). const unreachable = queues && target?.refusalCode === "unreachable"; @@ -158,7 +157,7 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread await client.queue({ ...input, ...(unreachable || agentSelection === null ? {} : { agent: agentSelection }), - mode: busy ? mode : "when-done", + mode: working ? mode : "when-done", }); } else if (agentSelection !== null) { await client.start({ ...input, agent: agentSelection }); @@ -288,7 +287,7 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread }} /> )} - {busy ? ( + {working ? ( void start()} data-testid="thread-move-confirm"> {starting ? "Moving…" - : busy + : working ? mode === "stop-now" ? "Stop and move" - : "Move when done" + : "Move when it finishes" : unreachable ? "Move when it connects" : "Move"} @@ -337,8 +336,9 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread } /** - * A busy thread's two ways to move (Part 3b): once it is free, or now, after - * its agent session ends. Says first what it is busy with. + * The two ways to move a thread whose agent is still at it (Part 3b): once + * the agent finishes, or now, after it is stopped. Says first what the agent + * is doing. */ function MoveWhenField({ label, @@ -360,18 +360,18 @@ function MoveWhenField({ }> = [ { value: "when-done", - label: "Move when it's done", - detail: describeThreadMoveWhenDone(label, waits), + label: "Move when the agent finishes", + detail: `Waits for the agent to finish, then moves to ${label}.`, }, { value: "stop-now", - label: "Stop and move now", - detail: "Stops the agent and anything it left running, then moves.", + label: "Stop the agent and move now", + detail: "Stops the agent and anything it started, then moves.", }, ]; return (
-

+

{describeThreadMoveWaits(waits)}

+ const callInFlight = (kind: PendingMoveCall["kind"]) => call !== null && call.kind === kind && call.threadId === threadId && call.requestId === requestId; - const cancelling = busyWith("cancel"); - const retrying = busyWith("retry"); + const cancelling = callInFlight("cancel"); + const retrying = callInFlight("retry"); return useMemo( () => pendingMove === null @@ -354,7 +354,7 @@ export function useMoveNotStartedNotice(input: { { keys: [] }, MoveNotStartedDismissalsSchema, ); - const busy = Boolean(thread?.move || thread?.pendingMove); + const anotherMove = Boolean(thread?.move || thread?.pendingMove); const activities = thread?.activities; const latest = useMemo(() => { if (!activities) return null; @@ -378,7 +378,7 @@ export function useMoveNotStartedNotice(input: { }, [dismissalKey, setDismissals]); return useMemo( () => - latest === null || busy || dismissed + latest === null || anotherMove || dismissed ? null : { id: `thread-move-not-started:${latest.requestId}`, @@ -388,6 +388,6 @@ export function useMoveNotStartedNotice(input: { dismissLabel: `Dismiss why it didn't move to ${latest.targetLabel}`, onDismiss: dismiss, }, - [busy, dismiss, dismissed, latest], + [anotherMove, dismiss, dismissed, latest], ); } diff --git a/apps/web/src/threadMove.ts b/apps/web/src/threadMove.ts index 3aac6082f..0841a86dc 100644 --- a/apps/web/src/threadMove.ts +++ b/apps/web/src/threadMove.ts @@ -231,35 +231,60 @@ export function threadMoveAgentSelection( // A move that waits for the thread to be free (docs/design/fleet.md, Part 3b) // --------------------------------------------------------------------------- -/** What a busy thread is doing, as it reads after "This thread is busy: ". */ -const WAIT_DOING: Record = { - turn: "its agent is working on a turn", - approval: "it's waiting for your approval", - question: "it's waiting for your answer", - side_turn: "a side answer is running", - background: "background work is still running", - command_run: "a command is running", - workflow_run: "a workflow is running", - voice: "a voice chat is on", - shared_checkout: "another thread is working in the same folder", +/** + * What the thread is doing, in the user's words, by kind: each says it on + * its own (`first`), and after a sentence about the agent (`after`), where + * "it" is the agent. Listed in the order they're told. + */ +const WAIT_SENTENCES = { + reply: { first: "The agent is still working on its reply.", after: null }, + answer: { + first: "The agent is waiting for your answer.", + after: "It's also waiting for your answer.", + }, + started: { + first: "Something the agent started is still running.", + after: "Something it started is still running too.", + }, + voice: { first: "A voice chat is still on.", after: "A voice chat is still on too." }, + folder: { + first: "Another thread is working in the same folder.", + after: "Another thread is working in the same folder too.", + }, +} as const; + +type WaitKind = keyof typeof WAIT_SENTENCES; + +const WAIT_KIND: Record = { + turn: "reply", + side_turn: "reply", + approval: "answer", + question: "answer", + background: "started", + command_run: "started", + workflow_run: "started", + voice: "voice", + shared_checkout: "folder", }; -/** What a busy thread is doing, in one line: "This thread is busy: its agent is working on a turn." */ -export function describeThreadMoveWaits(waits: ReadonlyArray): string { - const doing = [...new Set(waits.map((wait) => WAIT_DOING[wait.code]))]; - const last = doing.at(-1); - if (last === undefined) return "This thread is free."; - const list = doing.length === 1 ? last : `${doing.slice(0, -1).join(", ")} and ${last}`; - return `This thread is busy: ${list}.`; -} +const AGENT_KINDS: ReadonlySet = new Set(["reply", "answer", "started"]); -/** When a move that waits starts: "Moves to Mac mini when this turn ends." */ -export function describeThreadMoveWhenDone( - label: string, - waits: ReadonlyArray, -): string { - const first = waits[0]; - return first ? `Moves to ${label} ${first.words}.` : `Moves to ${label} as soon as it's free.`; +/** + * What the thread is doing that a move waits for, in at most two short + * sentences: "The agent is still working on its reply. Something it + * started is still running too." + */ +export function describeThreadMoveWaits(waits: ReadonlyArray): string { + const present = new Set(waits.map((wait) => WAIT_KIND[wait.code])); + const kinds = (Object.keys(WAIT_SENTENCES) as WaitKind[]).filter((kind) => present.has(kind)); + const [first, second] = kinds; + if (first === undefined) return ""; + const lead = WAIT_SENTENCES[first].first; + if (second === undefined) return lead; + const follow = AGENT_KINDS.has(first) + ? (WAIT_SENTENCES[second].after ?? WAIT_SENTENCES[second].first) + : WAIT_SENTENCES[second].first; + return `${lead} ${follow}`; } /** Why a thread whose move waits takes no new message: the send button's words. */ From f8f20515b9112749863373b4ee0933e99db03862 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 23:43:19 -0500 Subject: [PATCH 35/48] feat(web): the thread's title in the header opens its menu A click, tap, Enter or Space on the title opens under it the same menu the sidebar row opens on a right-click, so Move to another computer, Rename, Pin and the rest are one step from the open thread. The browser's own menu now takes focus for the keyboard, walks its items with the arrow keys and gives focus back when it closes. --- .../ChatView.move-thread.browser.tsx | 55 ++++++++++++ apps/web/src/components/ChatView.tsx | 7 ++ apps/web/src/components/chat/ChatHeader.tsx | 26 ++++-- .../src/components/chat/ThreadTitleMenu.tsx | 83 +++++++++++++++++++ apps/web/src/contextMenuFallback.test.ts | 44 ++++++++++ apps/web/src/contextMenuFallback.ts | 36 +++++++- 6 files changed, 243 insertions(+), 8 deletions(-) create mode 100644 apps/web/src/components/chat/ThreadTitleMenu.tsx diff --git a/apps/web/src/components/ChatView.move-thread.browser.tsx b/apps/web/src/components/ChatView.move-thread.browser.tsx index 8bd9ae1a5..07dd60a34 100644 --- a/apps/web/src/components/ChatView.move-thread.browser.tsx +++ b/apps/web/src/components/ChatView.move-thread.browser.tsx @@ -445,6 +445,61 @@ describe("ChatView (full app): moving a thread to another computer", () => { } }); + it("opens the thread's menu from its title in the header, by click or keyboard", async () => { + setPrimaryLinksSnapshot( + fleetLinksSnapshot({ + outbound: [fleetOutboundLink(FLEET_OFFLINE_ENVIRONMENT_ID, "Mac mini")], + }), + ); + const previews: Array = []; + const mounted = await mountChatView({ + viewport: WIDE_FOOTER_VIEWPORT, + snapshot: createSnapshotForTargetUser({ + targetMessageId: "msg-user-move-title" as MessageId, + targetText: "move from title", + }), + configureFixture: withThreadMoveFixture, + resolveRpc: (body) => { + if (body._tag === WS_METHODS.movesPreview) { + previews.push(body); + return FLEET_MOVE_PREVIEW; + } + return undefined; + }, + }); + + try { + await waitForServerConfigToApply(); + await saveMacMini("connected"); + await vi.waitFor(() => { + expect(readThreadMoveOffered(LOCAL_ENVIRONMENT_ID, undefined)).toBe(true); + }); + const title = page.getByRole("button", { name: `${THREAD_TITLE}, thread menu` }); + + // From the keyboard: Enter opens it, the down arrow reaches its first + // item, and Escape closes it and hands focus back to the title. + title.element().focus(); + await userEvent.keyboard("{Enter}"); + await expect.element(title).toHaveAttribute("aria-expanded", "true"); + await userEvent.keyboard("{ArrowDown}"); + await expect.element(page.getByRole("button", { name: "Rename" })).toHaveFocus(); + await userEvent.keyboard("{Escape}"); + await expect.element(title).toHaveFocus(); + await expect.element(title).toHaveAttribute("aria-expanded", "false"); + + // A click opens the same menu the sidebar row does, and its move opens the dialog. + await title.click(); + await expect.element(title).toHaveAttribute("aria-expanded", "true"); + await page.getByRole("button", { name: "Move to another computer" }).click(); + await expect + .element(page.getByTestId("thread-move-dialog").getByTestId("thread-move-target")) + .toMatchTextContent("Mac mini"); + expect(previews).toEqual([expect.objectContaining({ threadId: THREAD_ID })]); + } finally { + await mounted.cleanup(); + } + }); + it("offers a thread whose agent is still working two ways to move, and queues the one picked", async () => { setPrimaryLinksSnapshot( fleetLinksSnapshot({ diff --git a/apps/web/src/components/ChatView.tsx b/apps/web/src/components/ChatView.tsx index 743521f89..544128973 100644 --- a/apps/web/src/components/ChatView.tsx +++ b/apps/web/src/components/ChatView.tsx @@ -8712,6 +8712,13 @@ export default function ChatView(props: ChatViewProps) { onOpenParentThread={onOpenParentThread} backgroundRuns={backgroundRuns} activeThreadRef={activeThreadRef} + // The title opens the menu the thread's sidebar row opens: only where + // a row offers one (a saved, unarchived project thread). + threadMenuRef={ + isServerThread && !isGeneralChatThread && activeThread.archivedAt === null + ? activeThreadRef + : null + } onRunProjectScript={runProjectScript} onAddProjectScript={saveProjectScript} onUpdateProjectScript={updateProjectScript} diff --git a/apps/web/src/components/chat/ChatHeader.tsx b/apps/web/src/components/chat/ChatHeader.tsx index f891c038d..f144b2015 100644 --- a/apps/web/src/components/chat/ChatHeader.tsx +++ b/apps/web/src/components/chat/ChatHeader.tsx @@ -27,6 +27,7 @@ import ProjectScriptsControl, { type NewProjectScriptInput } from "../ProjectScr import { Toggle } from "../ui/toggle"; import { SidebarOpenTrigger } from "../ui/sidebar"; import { ProjectCrumbMenu } from "./ProjectCrumbMenu"; +import { ThreadTitleMenu } from "./ThreadTitleMenu"; import { openActiveFileViewer } from "../../fileViewerStore"; import { usePrimaryEnvironmentId } from "../../environments/primary"; import { ThreadActivityPopover, type ThreadTaskProgressState } from "./ThreadActivityPopover"; @@ -121,6 +122,12 @@ interface ChatHeaderProps { backgroundRuns: ReadonlyArray; /** The thread the background runs belong to; where their output is read from. */ activeThreadRef: ScopedThreadRef | null; + /** + * The thread whose menu the title opens (the one its sidebar row opens on a + * right-click). Null where no row would offer one: a draft, a General Chat, + * an archived thread. The title is then plain text. + */ + threadMenuRef?: ScopedThreadRef | null | undefined; onRunProjectScript: (script: ProjectScript) => void; onAddProjectScript: (input: NewProjectScriptInput) => Promise; onUpdateProjectScript: (scriptId: string, input: NewProjectScriptInput) => Promise; @@ -304,6 +311,7 @@ export const ChatHeader = memo(function ChatHeader({ outsideOrigin = null, backgroundRuns, activeThreadRef, + threadMenuRef = null, onRunProjectScript, onAddProjectScript, onUpdateProjectScript, @@ -384,12 +392,18 @@ export const ChatHeader = memo(function ChatHeader({
)} -

- {activeThreadTitle} -

+ {threadMenuRef ? ( +

+ +

+ ) : ( +

+ {activeThreadTitle} +

+ )} {forkContext ? ( + {openAt !== null ? ( + + ) : null} + + ); +}); + +/** + * Opens the menu once, then reports it closed. Mounted only while the menu is + * asked for, so the header doesn't follow the inbox the rest of the time: the + * inbox says whether the menu offers Wrap up or Reopen, as it does on the row. + */ +function ThreadTitleMenuOpener({ + threadRef, + position, + onClosed, +}: { + readonly threadRef: ScopedThreadRef; + readonly position: MenuPosition; + readonly onClosed: () => void; +}) { + const threadMenu = useThreadMenu(); + const { entries } = useInboxThreads(); + const threadKey = scopedThreadKey(threadRef); + const entry = entries.find((candidate) => candidate.threadKey === threadKey) ?? null; + const isDone = entry?.isDone ?? null; + const canMarkDone = entry?.canMarkDone ?? null; + const opened = useRef(false); + useEffect(() => { + if (opened.current) return; + opened.current = true; + const inbox = isDone === null || canMarkDone === null ? null : { isDone, canMarkDone }; + void threadMenu.open({ threadRef, inbox }, position).finally(onClosed); + }, [canMarkDone, isDone, onClosed, position, threadMenu, threadRef]); + return null; +} diff --git a/apps/web/src/contextMenuFallback.test.ts b/apps/web/src/contextMenuFallback.test.ts index 56b4596b9..46304f953 100644 --- a/apps/web/src/contextMenuFallback.test.ts +++ b/apps/web/src/contextMenuFallback.test.ts @@ -81,6 +81,14 @@ class FakeElement { return matches; } + focus() { + (document as unknown as FakeDocument).activeElement = this; + } + + get isConnected() { + return this.parent !== null; + } + getBoundingClientRect() { const left = Number.parseInt(this.style.left ?? "0", 10) || 0; const top = Number.parseInt(this.style.top ?? "0", 10) || 0; @@ -116,12 +124,20 @@ class FakeBody extends FakeElement { class FakeDocument { body = new FakeBody(); + activeElement: FakeElement | null = null; private readonly listeners = new Map(); createElement(tagName: string) { return new FakeElement(tagName); } + dispatchEvent(event: FakeDomEvent) { + for (const listener of this.listeners.get(event.type) ?? []) { + listener(event); + } + return true; + } + addEventListener(type: string, listener: FakeListener) { const existing = this.listeners.get(type) ?? []; existing.push(listener); @@ -218,4 +234,32 @@ describe("showContextMenuFallback", () => { await expect(selectionPromise).resolves.toBe("rename:project-b"); }); + + it("takes focus for the keyboard, walks it with the arrows, and gives it back", async () => { + const fakeDocument = document as unknown as FakeDocument; + const opener = new FakeElement("button"); + fakeDocument.body.appendChild(opener); + opener.focus(); + const selectionPromise = showContextMenuFallback([ + { id: "rename", label: "Rename" }, + { id: "pin", label: "Pin", disabled: true }, + { id: "move", label: "Move to another computer" }, + ]); + + // The menu holds focus, with no item picked, until an arrow picks one. + const menu = findButton("Rename")?.parent; + expect(fakeDocument.activeElement).toBe(menu); + const press = (key: string) => fakeDocument.dispatchEvent(new FakeDomEvent("keydown", { key })); + press("ArrowDown"); + expect(fakeDocument.activeElement).toBe(findButton("Rename")); + // Past the item that takes no press, and round from the last to the first. + press("ArrowDown"); + expect(fakeDocument.activeElement).toBe(findButton("Move to another computer")); + press("ArrowDown"); + expect(fakeDocument.activeElement).toBe(findButton("Rename")); + + press("Escape"); + await expect(selectionPromise).resolves.toBeNull(); + expect(fakeDocument.activeElement).toBe(opener); + }); }); diff --git a/apps/web/src/contextMenuFallback.ts b/apps/web/src/contextMenuFallback.ts index ebca1ba31..ca911f9d7 100644 --- a/apps/web/src/contextMenuFallback.ts +++ b/apps/web/src/contextMenuFallback.ts @@ -36,6 +36,11 @@ export function showContextMenuFallback( // a press has started on the menu itself, or to a keyboard's Enter or // Space (a click with no press: `detail` 0). let pressedOnMenu = !coarse; + // Focus goes into the menu, so a keyboard can reach its items, and back + // to whatever opened it once it closes. + const opener = document.activeElement as HTMLElement | null; + // Each open menu's items that take a press, in order, for the arrow keys. + const itemsByMenu = new Map(); const cleanup = (result: T | null) => { document.removeEventListener("keydown", onKeyDown); @@ -43,6 +48,7 @@ export function showContextMenuFallback( for (const menu of menuStack) { menu.remove(); } + if (opener?.isConnected) opener.focus({ preventScroll: true }); resolve(result); }; @@ -50,7 +56,26 @@ export function showContextMenuFallback( if (event.key === "Escape") { event.preventDefault(); cleanup(null); + return; } + if (event.key !== "ArrowDown" && event.key !== "ArrowUp") return; + // Up and down walk the items of the menu that has focus (the top one). + const focused = document.activeElement as HTMLButtonElement | null; + const items = + menuStack + .map((menu) => itemsByMenu.get(menu) ?? []) + .find((candidate) => focused !== null && candidate.includes(focused)) ?? + (menuStack[0] ? (itemsByMenu.get(menuStack[0]) ?? []) : []); + if (items.length === 0) return; + event.preventDefault(); + const at = focused === null ? -1 : items.indexOf(focused); + const next = + at === -1 + ? event.key === "ArrowDown" + ? 0 + : items.length - 1 + : (at + (event.key === "ArrowDown" ? 1 : items.length - 1)) % items.length; + items[next]?.focus({ preventScroll: true }); }; const closeMenusFromLevel = (level: number) => { @@ -68,10 +93,13 @@ export function showContextMenuFallback( closeMenusFromLevel(level); const menu = document.createElement("div"); - menu.className = `fixed z-[10000] ${coarse ? "min-w-[200px]" : "min-w-[160px]"} rounded-md border border-border bg-popover py-1 shadow-xl animate-in fade-in zoom-in-95`; + menu.className = `fixed z-[10000] ${coarse ? "min-w-[200px]" : "min-w-[160px]"} rounded-md border border-border bg-popover py-1 shadow-xl outline-none animate-in fade-in zoom-in-95`; menu.style.left = `${preferredLeft}px`; menu.style.top = `${preferredTop}px`; menu.dataset.level = String(level); + menu.tabIndex = -1; + const focusable: HTMLButtonElement[] = []; + itemsByMenu.set(menu, focusable); for (const item of entries) { // A new group gets a divider above it, never at the top of the menu. @@ -88,7 +116,7 @@ export function showContextMenuFallback( button.disabled = isDisabled; button.className = isDisabled ? `flex w-full items-center gap-2 ${itemSize} text-left text-muted-foreground/60 cursor-not-allowed` - : `flex w-full items-center gap-2 ${itemSize} text-left text-popover-foreground hover:bg-accent cursor-default`; + : `flex w-full items-center gap-2 ${itemSize} text-left text-popover-foreground hover:bg-accent focus-visible:bg-accent outline-none cursor-default`; const label = document.createElement("span"); label.className = "min-w-0 flex-1 truncate"; @@ -103,6 +131,7 @@ export function showContextMenuFallback( } if (!isDisabled) { + focusable.push(button); if (hasChildren) { button.addEventListener("mouseenter", () => { const rect = button.getBoundingClientRect(); @@ -148,6 +177,9 @@ export function showContextMenuFallback( requestAnimationFrame(() => { clampMenuPosition(menu, preferredLeft, preferredTop); }); + // The menu itself takes focus, not an item: nothing looks picked under a + // mouse, and the arrows or Tab go on to the items. + if (level === 0) menu.focus({ preventScroll: true }); }; // A new press outside closes the menu. On a touch screen that is a From 4bcddd5bb1bfb99ad499102132d24736cb340234 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Sun, 15 Nov 2026 23:50:15 -0500 Subject: [PATCH 36/48] fix(web): stop and move now isn't offered while a command runs in the user's terminal The server refuses that move, since nothing here can stop a command in the user's own terminal. The dialog now greys out the choice, says why in the server's words, and keeps "Move when the agent finishes" picked. --- .../ChatView.move-thread.browser.tsx | 32 +++++++++++++-- .../components/ThreadMoveDialogContent.tsx | 39 ++++++++++++++++--- apps/web/src/threadMove.ts | 12 ++++++ 3 files changed, 74 insertions(+), 9 deletions(-) diff --git a/apps/web/src/components/ChatView.move-thread.browser.tsx b/apps/web/src/components/ChatView.move-thread.browser.tsx index 07dd60a34..0a314a288 100644 --- a/apps/web/src/components/ChatView.move-thread.browser.tsx +++ b/apps/web/src/components/ChatView.move-thread.browser.tsx @@ -508,6 +508,10 @@ describe("ChatView (full app): moving a thread to another computer", () => { ); const macMini = FLEET_MOVE_PREVIEW.targets[1]!; const queues: Array = []; + let waitsFor: NonNullable = [ + { code: "turn", words: "when this turn ends" }, + { code: "background", words: "when its background work finishes" }, + ]; const mounted = await mountChatView({ viewport: WIDE_FOOTER_VIEWPORT, snapshot: createSnapshotForTargetUser({ @@ -519,10 +523,7 @@ describe("ChatView (full app): moving a thread to another computer", () => { if (body._tag === WS_METHODS.movesPreview) { return { ...FLEET_MOVE_PREVIEW, - waitsFor: [ - { code: "turn", words: "when this turn ends" }, - { code: "background", words: "when its background work finishes" }, - ], + waitsFor, targets: [macMini], } satisfies ThreadMovePreview; } @@ -583,6 +584,29 @@ describe("ChatView (full app): moving a thread to another computer", () => { mode: "stop-now", }), ]); + + // A command runs in the user's terminal, which nothing here stops: + // stopping isn't offered, says why, and waiting stays picked. + waitsFor = [{ code: "command_run", words: "when its command finishes running" }]; + await openCommandPaletteFromTrigger(); + await page + .getByTestId("command-palette") + .getByText("Move to another computer", { exact: true }) + .click(); + await expect + .element(dialog.getByRole("radio", { name: /Stop the agent and move now/u })) + .toBeDisabled(); + await expect + .element(dialog.getByTestId("thread-move-mode-stop-now-detail")) + .toHaveTextContent( + "A command from this thread is running in your terminal. Stop it there, or move the thread when it's done.", + ); + await expect + .element(dialog.getByRole("radio", { name: /Move when the agent finishes/u })) + .toBeChecked(); + await expect + .element(dialog.getByTestId("thread-move-confirm")) + .toHaveTextContent("Move when it finishes"); } finally { await mounted.cleanup(); } diff --git a/apps/web/src/components/ThreadMoveDialogContent.tsx b/apps/web/src/components/ThreadMoveDialogContent.tsx index 37f7cfd71..561080a2a 100644 --- a/apps/web/src/components/ThreadMoveDialogContent.tsx +++ b/apps/web/src/components/ThreadMoveDialogContent.tsx @@ -25,6 +25,7 @@ import { useProjectSetupComputerReady, useProjectSetupTargets, } from "../environments/projectSetups/projectSetupTargets"; +import { cn } from "../lib/utils"; import { selectSidebarThreadSummaryByRef, useStore } from "../store"; import { describeThreadMoveWaits, @@ -34,6 +35,7 @@ import { pickDefaultMoveTarget, threadMoveAgentKey, threadMoveAgentSelection, + threadMoveStopNowRefusal, type ThreadMoveAgentChoice, } from "../threadMove"; import { openProjectSetupDialog } from "./ProjectSetupDialog"; @@ -80,7 +82,7 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread const [previewState, setPreviewState] = useState({ status: "loading" }); const [targetId, setTargetId] = useState(null); const [agent, setAgent] = useState(null); - const [mode, setMode] = useState("when-done"); + const [pickedMode, setMode] = useState("when-done"); const [starting, setStarting] = useState(false); const [startError, setStartError] = useState(null); // Asked again when a computer that lacked the project has been set up. @@ -133,6 +135,9 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread // Still working: the move can wait for the agent to finish, or stop it first. const waits = queues ? (preview?.waitsFor ?? []) : []; const working = waits.length > 0; + // Stopping isn't offered while a command runs in the user's terminal. + const stopNowRefusal = threadMoveStopNowRefusal(waits); + const mode: ThreadPendingMoveMode = stopNowRefusal === null ? pickedMode : "when-done"; // Out of reach: a queued move to it waits for it to connect, and runs // with the agent it suggests then (no agent sent). const unreachable = queues && target?.refusalCode === "unreachable"; @@ -292,6 +297,7 @@ export function ThreadMoveDialogContent({ threadRef }: { threadRef: ScopedThread label={target.label} waits={waits} mode={mode} + stopNowRefusal={stopNowRefusal} disabled={starting} onChange={setMode} /> @@ -344,12 +350,15 @@ function MoveWhenField({ label, waits, mode, + stopNowRefusal, disabled, onChange, }: { readonly label: string; readonly waits: ReadonlyArray; readonly mode: ThreadPendingMoveMode; + /** Why stopping isn't offered now (threadMoveStopNowRefusal); null when it is. */ + readonly stopNowRefusal: string | null; readonly disabled: boolean; readonly onChange: (mode: ThreadPendingMoveMode) => void; }) { @@ -357,16 +366,20 @@ function MoveWhenField({ readonly value: ThreadPendingMoveMode; readonly label: string; readonly detail: string; + /** Why it can't be picked, under its own words; null when it can. */ + readonly refusal: string | null; }> = [ { value: "when-done", label: "Move when the agent finishes", detail: `Waits for the agent to finish, then moves to ${label}.`, + refusal: null, }, { value: "stop-now", label: "Stop the agent and move now", detail: "Stops the agent and anything it started, then moves.", + refusal: stopNowRefusal, }, ]; return ( @@ -384,13 +397,29 @@ function MoveWhenField({ {choices.map((choice) => ( ))} diff --git a/apps/web/src/threadMove.ts b/apps/web/src/threadMove.ts index 0841a86dc..85431d2b8 100644 --- a/apps/web/src/threadMove.ts +++ b/apps/web/src/threadMove.ts @@ -287,6 +287,18 @@ export function describeThreadMoveWaits(waits: ReadonlyArray): s return `${lead} ${follow}`; } +/** + * Why "Stop the agent and move now" isn't offered, or null when it is: a + * command from the thread runs in the user's own terminal, which nothing + * here stops. The server refuses that move with these same words + * (threadMoveDecisions.ts, decideMoveQueue). + */ +export function threadMoveStopNowRefusal(waits: ReadonlyArray): string | null { + return waits.some((wait) => wait.code === "command_run") + ? "A command from this thread is running in your terminal. Stop it there, or move the thread when it's done." + : null; +} + /** Why a thread whose move waits takes no new message: the send button's words. */ export function pendingMoveSendBlockReason(pendingMove: ThreadPendingMove): string { return `This thread is waiting to move to ${pendingMove.target.label}. Cancel the move to keep working here.`; From 3b31829cbc821251fbe60f2d7cc7ffa475063a1b Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Mon, 16 Nov 2026 00:35:43 -0500 Subject: [PATCH 37/48] feat(desktop): the window follows the cursor while the page says a drag is on Header controls that are also buttons can't be part of the window's drag area, and Electron can't start a native drag from page content. A new bridge call, setWindowDragging, starts and stops a loop that keeps the window at the cursor's starting offset, about 60 times a second, by position only. It runs only during a drag and ends on stop, on blur, or after 10 s. A maximized or full-screen window stays put. --- apps/desktop/src/ipc/DesktopIpcHandlers.ts | 2 + apps/desktop/src/ipc/channels.ts | 1 + apps/desktop/src/ipc/methods/window.ts | 12 ++ apps/desktop/src/main.ts | 3 + apps/desktop/src/preload.ts | 2 + .../src/window/DesktopWindowDrag.test.ts | 132 +++++++++++++++++ apps/desktop/src/window/DesktopWindowDrag.ts | 138 ++++++++++++++++++ packages/contracts/src/ipc.ts | 6 + 8 files changed, 296 insertions(+) create mode 100644 apps/desktop/src/window/DesktopWindowDrag.test.ts create mode 100644 apps/desktop/src/window/DesktopWindowDrag.ts diff --git a/apps/desktop/src/ipc/DesktopIpcHandlers.ts b/apps/desktop/src/ipc/DesktopIpcHandlers.ts index b33cd036b..148f3e925 100644 --- a/apps/desktop/src/ipc/DesktopIpcHandlers.ts +++ b/apps/desktop/src/ipc/DesktopIpcHandlers.ts @@ -51,6 +51,7 @@ import { setTheme, getPowerState, setBackdropActive, + setWindowDragging, showContextMenu, } from "./methods/window.ts"; import { @@ -158,6 +159,7 @@ export const installDesktopIpcHandlers = Effect.gen(function* () { yield* ipc.handle(setTheme); yield* ipc.handle(getPowerState); yield* ipc.handle(setBackdropActive); + yield* ipc.handle(setWindowDragging); yield* ipc.handle(setTaskbarStatus); yield* ipc.handle(getBackgroundOptions); yield* ipc.handle(setBackgroundOptions); diff --git a/apps/desktop/src/ipc/channels.ts b/apps/desktop/src/ipc/channels.ts index 75789c696..fd7e2caf2 100644 --- a/apps/desktop/src/ipc/channels.ts +++ b/apps/desktop/src/ipc/channels.ts @@ -46,6 +46,7 @@ export const SET_THEME_CHANNEL = "desktop:set-theme"; export const GET_POWER_STATE_CHANNEL = "desktop:get-power-state"; export const POWER_STATE_CHANNEL = "desktop:power-state"; export const SET_BACKDROP_ACTIVE_CHANNEL = "desktop:set-backdrop-active"; +export const SET_WINDOW_DRAGGING_CHANNEL = "desktop:set-window-dragging"; export const BACKDROP_TOO_HEAVY_CHANNEL = "desktop:backdrop-too-heavy"; export const RESET_THEME_CHANNEL = "desktop:reset-theme"; export const SET_TASKBAR_STATUS_CHANNEL = "desktop:set-taskbar-status"; diff --git a/apps/desktop/src/ipc/methods/window.ts b/apps/desktop/src/ipc/methods/window.ts index 1a2258a17..5685d8ea2 100644 --- a/apps/desktop/src/ipc/methods/window.ts +++ b/apps/desktop/src/ipc/methods/window.ts @@ -26,6 +26,7 @@ import * as DesktopStatusIndicator from "../../window/DesktopStatusIndicator.ts" import * as DesktopBackdropMonitor from "../../window/DesktopBackdropMonitor.ts"; import * as DesktopThemeAppearance from "../../window/DesktopThemeAppearance.ts"; import * as DesktopWindow from "../../window/DesktopWindow.ts"; +import * as DesktopWindowDrag from "../../window/DesktopWindowDrag.ts"; const ContextMenuPosition = Schema.Struct({ x: Schema.Number, @@ -233,3 +234,14 @@ export const setBackdropActive = makeIpcMethod({ yield* monitor.setBackdropActive(active); }), }); + +/** A press on a header control became a drag (true), or ended (false): the window follows the cursor. */ +export const setWindowDragging = makeIpcMethod({ + channel: IpcChannels.SET_WINDOW_DRAGGING_CHANNEL, + payload: Schema.Boolean, + result: Schema.Void, + handler: Effect.fn("desktop.ipc.window.setWindowDragging")(function* (dragging) { + const windowDrag = yield* DesktopWindowDrag.DesktopWindowDrag; + yield* windowDrag.setDragging(dragging); + }), +}); diff --git a/apps/desktop/src/main.ts b/apps/desktop/src/main.ts index 05b590742..4dfac0c24 100644 --- a/apps/desktop/src/main.ts +++ b/apps/desktop/src/main.ts @@ -65,6 +65,7 @@ import { runAppImageCommandIfAsked } from "./cliCommand/appImageCommand.ts"; import * as DesktopCliCommand from "./cliCommand/DesktopCliCommand.ts"; import * as DesktopUpdates from "./updates/DesktopUpdates.ts"; import * as DesktopBackdropMonitor from "./window/DesktopBackdropMonitor.ts"; +import * as DesktopWindowDrag from "./window/DesktopWindowDrag.ts"; import * as DesktopThemeAppearance from "./window/DesktopThemeAppearance.ts"; import * as DesktopWindow from "./window/DesktopWindow.ts"; import * as DesktopStartupFailurePrompt from "./window/DesktopStartupFailurePrompt.ts"; @@ -307,6 +308,8 @@ const desktopWindowLayer = DesktopWindow.layer.pipe( // appearance changes; the art's power and CPU watch sits beside it. Layer.provideMerge(DesktopThemeAppearance.layer), Layer.provideMerge(DesktopBackdropMonitor.layer), + // Moving the window from header controls that are also buttons. + Layer.provideMerge(DesktopWindowDrag.layer), Layer.provideMerge(PreviewSession.layer), Layer.provideMerge(PreviewAutomation.layer), // Beneath the window and the automation, which both ask whose a page is. diff --git a/apps/desktop/src/preload.ts b/apps/desktop/src/preload.ts index c99af6886..15211d32b 100644 --- a/apps/desktop/src/preload.ts +++ b/apps/desktop/src/preload.ts @@ -335,6 +335,8 @@ contextBridge.exposeInMainWorld("desktopBridge", { }, setBackdropActive: (active) => ipcRenderer.invoke(IpcChannels.SET_BACKDROP_ACTIVE_CHANNEL, active), + setWindowDragging: (dragging) => + ipcRenderer.invoke(IpcChannels.SET_WINDOW_DRAGGING_CHANNEL, dragging), onBackdropTooHeavy: (listener) => { const wrappedListener = () => listener(); ipcRenderer.on(IpcChannels.BACKDROP_TOO_HEAVY_CHANNEL, wrappedListener); diff --git a/apps/desktop/src/window/DesktopWindowDrag.test.ts b/apps/desktop/src/window/DesktopWindowDrag.test.ts new file mode 100644 index 000000000..dbddb5226 --- /dev/null +++ b/apps/desktop/src/window/DesktopWindowDrag.test.ts @@ -0,0 +1,132 @@ +import { assert, describe, it } from "@effect/vitest"; + +import { + DRAG_FRAME_MS, + DRAG_SAFETY_MS, + makeWindowDragLoop, + type DraggableWindow, +} from "./DesktopWindowDrag.ts"; + +/** A clock the test moves by hand: intervals and one-shot timers, in ms. */ +function makeClock() { + let now = 0; + let nextId = 0; + const timers = new Map void }>(); + return { + clock: { + setInterval: (run: () => void, ms: number) => { + nextId += 1; + timers.set(nextId, { at: now + ms, every: ms, run }); + return nextId; + }, + clearInterval: (handle: unknown) => void timers.delete(handle as number), + setTimeout: (run: () => void, ms: number) => { + nextId += 1; + timers.set(nextId, { at: now + ms, every: null, run }); + return nextId; + }, + clearTimeout: (handle: unknown) => void timers.delete(handle as number), + }, + advance(ms: number) { + const end = now + ms; + for (;;) { + const due = [...timers.entries()] + .filter(([, timer]) => timer.at <= end) + .toSorted(([, left], [, right]) => left.at - right.at)[0]; + if (!due) break; + const [id, timer] = due; + now = timer.at; + if (timer.every === null) timers.delete(id); + else timer.at += timer.every; + timer.run(); + } + now = end; + }, + pending: () => timers.size, + }; +} + +function makeWindow(input?: { maximized?: boolean; fullScreen?: boolean }) { + let position: [number, number] = [100, 200]; + const moves: Array<[number, number]> = []; + const blurListeners = new Set<() => void>(); + const window: DraggableWindow = { + getPosition: () => [...position], + setPosition: (x, y) => { + position = [x, y]; + moves.push([x, y]); + }, + isMaximized: () => input?.maximized ?? false, + isFullScreen: () => input?.fullScreen ?? false, + isDestroyed: () => false, + once: (_event, listener) => blurListeners.add(listener), + removeListener: (_event, listener) => blurListeners.delete(listener), + }; + return { + window, + moves, + blur: () => { + for (const listener of blurListeners) listener(); + }, + }; +} + +describe("makeWindowDragLoop", () => { + it("moves the window by how far the cursor moved, in whole pixels, only when it changed", () => { + const time = makeClock(); + let cursor = { x: 150, y: 210 }; + const loop = makeWindowDragLoop({ cursor: () => cursor, clock: time.clock }); + const target = makeWindow(); + + loop.start(target.window); + time.advance(DRAG_FRAME_MS * 3); + assert.deepStrictEqual(target.moves, []); + + cursor = { x: 180.4, y: 195.6 }; + time.advance(DRAG_FRAME_MS * 3); + assert.deepStrictEqual(target.moves, [[130, 186]]); + + loop.stop(); + cursor = { x: 400, y: 400 }; + time.advance(DRAG_FRAME_MS * 3); + assert.deepStrictEqual(target.moves, [[130, 186]]); + // Nothing runs once it stopped. + assert.strictEqual(time.pending(), 0); + }); + + it("stops when the window loses focus, or after a while with no stop", () => { + const time = makeClock(); + let cursor = { x: 0, y: 0 }; + const loop = makeWindowDragLoop({ cursor: () => cursor, clock: time.clock }); + + const blurred = makeWindow(); + loop.start(blurred.window); + blurred.blur(); + assert.isFalse(loop.isDragging()); + assert.strictEqual(time.pending(), 0); + + const forgotten = makeWindow(); + loop.start(forgotten.window); + time.advance(DRAG_SAFETY_MS); + assert.isFalse(loop.isDragging()); + assert.strictEqual(time.pending(), 0); + cursor = { x: 50, y: 50 }; + time.advance(DRAG_FRAME_MS * 3); + assert.deepStrictEqual(forgotten.moves, []); + }); + + it("leaves a maximized or full-screen window where it is", () => { + const time = makeClock(); + let cursor = { x: 0, y: 0 }; + const loop = makeWindowDragLoop({ cursor: () => cursor, clock: time.clock }); + + for (const target of [makeWindow({ maximized: true }), makeWindow({ fullScreen: true })]) { + loop.start(target.window); + assert.isFalse(loop.isDragging()); + cursor = { x: cursor.x + 40, y: cursor.y + 40 }; + time.advance(DRAG_FRAME_MS * 3); + assert.deepStrictEqual(target.moves, []); + } + assert.strictEqual(time.pending(), 0); + }); +}); diff --git a/apps/desktop/src/window/DesktopWindowDrag.ts b/apps/desktop/src/window/DesktopWindowDrag.ts new file mode 100644 index 000000000..be72c447e --- /dev/null +++ b/apps/desktop/src/window/DesktopWindowDrag.ts @@ -0,0 +1,138 @@ +/** + * Moving the window from a header control that is also a button (the + * project name and the thread title open menus on a click). Their + * buttons can't be part of the window's drag area, and Electron has no way + * to start a native drag from page content, so the app moves the window + * itself: the page says when a press turned into a drag (`setWindowDragging` + * on the desktop bridge) and when it ended. + * + * While a drag lasts, about 60 times a second the window follows the cursor, + * keeping the offset it had when the drag started. Whole pixels, position + * only (never `setBounds`), so the window can't change size, which matters + * on Windows across monitors with different scaling. The loop runs only + * during a drag: it ends on the page's stop, when the window loses focus, or + * after 10 s without a stop, in case one never comes. A maximized or + * full-screen window doesn't move. + */ +import * as Context from "effect/Context"; +import * as Effect from "effect/Effect"; +import * as Layer from "effect/Layer"; + +import * as Electron from "electron"; + +import * as ElectronWindow from "../electron/ElectronWindow.ts"; + +export const DRAG_FRAME_MS = 16; +/** A drag with no stop after this long ends on its own. */ +export const DRAG_SAFETY_MS = 10_000; + +interface Point { + readonly x: number; + readonly y: number; +} + +/** What the loop needs of a window. */ +export interface DraggableWindow { + readonly getPosition: () => number[]; + readonly setPosition: (x: number, y: number) => void; + readonly isMaximized: () => boolean; + readonly isFullScreen: () => boolean; + readonly isDestroyed: () => boolean; + readonly once: (event: "blur", listener: () => void) => unknown; + readonly removeListener: (event: "blur", listener: () => void) => unknown; +} + +export interface WindowDragClock { + readonly setInterval: (run: () => void, ms: number) => unknown; + readonly clearInterval: (handle: unknown) => void; + readonly setTimeout: (run: () => void, ms: number) => unknown; + readonly clearTimeout: (handle: unknown) => void; +} + +/** One drag at a time: `start` follows the cursor, `stop` ends it. */ +export function makeWindowDragLoop(input: { + readonly cursor: () => Point; + readonly clock: WindowDragClock; +}) { + const { cursor, clock } = input; + let current: { readonly stop: () => void } | null = null; + + const stop = () => { + current?.stop(); + current = null; + }; + + const start = (window: DraggableWindow) => { + stop(); + if (window.isDestroyed() || window.isMaximized() || window.isFullScreen()) return; + const [windowX = 0, windowY = 0] = window.getPosition(); + const from = cursor(); + const offset = { x: from.x - windowX, y: from.y - windowY }; + let last = { x: windowX, y: windowY }; + + const follow = () => { + if (window.isDestroyed()) { + stop(); + return; + } + const at = cursor(); + const next = { x: Math.round(at.x - offset.x), y: Math.round(at.y - offset.y) }; + if (next.x === last.x && next.y === last.y) return; + last = next; + window.setPosition(next.x, next.y); + }; + + const frame = clock.setInterval(follow, DRAG_FRAME_MS); + const safety = clock.setTimeout(stop, DRAG_SAFETY_MS); + const onBlur = () => stop(); + window.once("blur", onBlur); + current = { + stop: () => { + clock.clearInterval(frame); + clock.clearTimeout(safety); + if (!window.isDestroyed()) window.removeListener("blur", onBlur); + }, + }; + }; + + return { start, stop, isDragging: () => current !== null }; +} + +export interface DesktopWindowDragShape { + /** A press on a header control became a drag (true), or ended (false). */ + readonly setDragging: (dragging: boolean) => Effect.Effect; +} + +export class DesktopWindowDrag extends Context.Service()( + "threadlines/desktop/WindowDrag", +) {} + +export const layer = Layer.effect( + DesktopWindowDrag, + Effect.gen(function* () { + const electronWindow = yield* ElectronWindow.ElectronWindow; + const loop = makeWindowDragLoop({ + cursor: () => Electron.screen.getCursorScreenPoint(), + clock: { + setInterval: (run, ms) => setInterval(run, ms), + clearInterval: (handle) => clearInterval(handle as ReturnType), + setTimeout: (run, ms) => setTimeout(run, ms), + clearTimeout: (handle) => clearTimeout(handle as ReturnType), + }, + }); + yield* Effect.addFinalizer(() => Effect.sync(loop.stop)); + return DesktopWindowDrag.of({ + setDragging: (dragging) => + Effect.gen(function* () { + if (!dragging) { + loop.stop(); + return; + } + // The press happened in the window that has focus. + const window = yield* electronWindow.focusedMainOrFirst; + if (window._tag === "None") return; + loop.start(window.value); + }).pipe(Effect.withSpan("desktop.windowDrag.setDragging")), + }); + }), +); diff --git a/packages/contracts/src/ipc.ts b/packages/contracts/src/ipc.ts index 5a297e209..ba72ad813 100644 --- a/packages/contracts/src/ipc.ts +++ b/packages/contracts/src/ipc.ts @@ -1602,6 +1602,12 @@ export interface DesktopBridge { * watchdog samples the backdrop frame's process. */ setBackdropActive?: (active: boolean) => Promise; + /** + * A press on a header control that is also a button (the project name, + * the thread title) became a drag (true), or ended (false): the window + * follows the cursor meanwhile. Absent on older desktop builds. + */ + setWindowDragging?: (dragging: boolean) => Promise; /** The watchdog found the backdrop frame over its CPU budget; the app makes the art still. */ onBackdropTooHeavy?: (listener: () => void) => () => void; /** View › Reset Theme: the app goes back to the built-in System theme. */ From a6eb616e3cb9ac8389241697bdb7a0c9c291e6d1 Mon Sep 17 00:00:00 2001 From: Badcuban <108198679+badcuban@users.noreply.github.com> Date: Mon, 16 Nov 2026 00:35:43 -0500 Subject: [PATCH 38/48] feat(web): press and drag on the project name or thread title moves the desktop window A press that moves more than 4 px becomes a window drag in the desktop app, and its click is swallowed so the menu stays shut. A plain click opens the menu as before, and the keyboard is unchanged. Web and phone builds don't change. --- .../src/components/chat/ProjectCrumbMenu.tsx | 19 +++- .../src/components/chat/ThreadTitleMenu.tsx | 7 +- apps/web/src/hooks/useWindowDragPress.ts | 17 ++++ apps/web/src/lib/windowDragPress.test.ts | 87 +++++++++++++++++ apps/web/src/lib/windowDragPress.ts | 95 +++++++++++++++++++ 5 files changed, 222 insertions(+), 3 deletions(-) create mode 100644 apps/web/src/hooks/useWindowDragPress.ts create mode 100644 apps/web/src/lib/windowDragPress.test.ts create mode 100644 apps/web/src/lib/windowDragPress.ts diff --git a/apps/web/src/components/chat/ProjectCrumbMenu.tsx b/apps/web/src/components/chat/ProjectCrumbMenu.tsx index c13c019d5..5bf2242b4 100644 --- a/apps/web/src/components/chat/ProjectCrumbMenu.tsx +++ b/apps/web/src/components/chat/ProjectCrumbMenu.tsx @@ -1,10 +1,11 @@ import type { EditorId, EnvironmentId, ResolvedKeybindingsConfig } from "@threadlines/contracts"; import { areFilesystemPathsEqual, shortenHomePath } from "@threadlines/shared/path"; import { threadWorkingCwdLabel } from "@threadlines/shared/threadCwd"; -import { memo, useCallback, useEffect, useMemo } from "react"; +import { memo, type MouseEvent, useCallback, useEffect, useMemo, useState } from "react"; import { ChevronDownIcon, CopyIcon, FolderGit2Icon } from "lucide-react"; import { isOpenFavoriteEditorShortcut, shortcutLabelForCommand } from "../../keybindings"; import { usePreferredEditor } from "../../editorPreferences"; +import { useWindowDragPress } from "../../hooks/useWindowDragPress"; import { Menu, MenuGroup, @@ -116,11 +117,25 @@ export const ProjectCrumbMenu = memo(function ProjectCrumbMenu({ ); + // In the desktop app, press and drag moves the window instead. The press + // then doesn't open the menu on mousedown, so a click (not a key, which + // the trigger handles itself) opens it here. + const dragPress = useWindowDragPress(); + const [open, setOpen] = useState(false); + return ( - + ) => { + if (event.detail > 0) setOpen((current) => !current); + }, + } + : {})} type="button" className="-ms-1 inline-flex min-w-0 max-w-48 items-center gap-0.5 rounded-md px-1 py-0.5 text-sm text-muted-foreground outline-none transition-colors hover:bg-accent hover:text-foreground focus-visible:ring-2 focus-visible:ring-ring data-popup-open:bg-accent data-popup-open:text-foreground" aria-label={`${projectName}, project options`} diff --git a/apps/web/src/components/chat/ThreadTitleMenu.tsx b/apps/web/src/components/chat/ThreadTitleMenu.tsx index 0c6343f16..63834aaae 100644 --- a/apps/web/src/components/chat/ThreadTitleMenu.tsx +++ b/apps/web/src/components/chat/ThreadTitleMenu.tsx @@ -3,7 +3,8 @@ * Enter, Space, a tap) opens under the title the same menu its sidebar row * opens on a right-click (useThreadMenu), with Rename, Pin, Move to another * computer, Archive and the rest. Wherever the header sits: its own row, the - * window's title row (titleBarSlot), or a phone. + * window's title row (titleBarSlot), or a phone. In the desktop app a press + * that drags moves the window instead (useWindowDragPress). * * @module ThreadTitleMenu */ @@ -14,6 +15,7 @@ import { memo, useCallback, useEffect, useRef, useState } from "react"; import { useInboxThreads } from "../../hooks/useInboxThreads"; import { useThreadMenu } from "../../hooks/useThreadMenu"; +import { useWindowDragPress } from "../../hooks/useWindowDragPress"; type MenuPosition = { readonly x: number; readonly y: number }; @@ -26,9 +28,12 @@ export const ThreadTitleMenu = memo(function ThreadTitleMenu({ }) { const [openAt, setOpenAt] = useState(null); const closed = useCallback(() => setOpenAt(null), []); + // In the desktop app, press and drag moves the window instead. + const dragPress = useWindowDragPress(); return ( <>