Skip to content

ACP quit and reopen drops active turns, prompts, and subagents across shutdown and startup recovery #36957

Closed as not planned

Description

@mahmoodnizamani94-png

Quitting and reopening T3 Code while ACP sessions are running drops active turns, loses newly submitted prompts and attachments, and kills native subagents even when continueThreadsAfterServerUpdate is on. Tracing shutdown and startup recovery through packages/provider-acp/src/server/adapter.ts, apps/server/src/serverRuntimeStartup.ts, apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts, apps/server/src/orchestration-v2/ProjectionStore.ts, and apps/server/src/orchestration-v2/RestartContinuation.ts shows five places where the handoff breaks.

During server.shutdown in apps/server/src/serverRuntimeStartup.ts, AcpAdapterV2 in packages/provider-acp/src/server/adapter.ts has no shutdown guard on handleRuntimeTerminationAtGeneration. Whenever an ACP child exits during app quit before shutdown reconciliation finishes, handleRuntimeTerminationAtGeneration marks every running entry in nativeChildren as failed and calls finalizeTurn with failed and transport_error. Since restartContinuationBlocked in apps/server/src/orchestration-v2/RestartContinuation.ts skips any run whose latest provider turn is failed, that teardown race kills active turns and subagents before restart continuation even runs.

On top of that, ProviderRuntimeRecoveryService.prepareForShutdown in apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts runs its Effect.forEach handoff with { concurrency: 2, discard: true }, so when several ACP threads are open, the ones waiting behind that cap get starved while child processes are already exiting.

Then on startup, ProjectionStore.getRecoveryThreadIds for "restart-continuations" in apps/server/src/orchestration-v2/ProjectionStore.ts only selects orchestration_v2_projection_runs matching WHERE status = 'starting' AND CASE WHEN json_valid(payload_json) THEN json_extract(payload_json, '$.restartContinuationOfRunId') IS NOT NULL ELSE 0 END once orchestration_v2_projection_provider_sessions rows are stopped, error, or missing. Any thread whose session row already flipped during teardown or had not bound yet drops preparing, starting, running, and waiting runs on boot.

Inside restartContinuationRun in apps/server/src/orchestration-v2/RestartContinuation.ts, !neverStarted(candidate) from apps/server/src/orchestration-v2/RestartBackgroundNote.ts drops runs where startedAt === null and requires an already bound active providerThread with a strong nativeThreadRef, which throws away turns caught during workspace prep or initial session bind. And when continueRestartedRun runs for a turn interrupted before the provider committed the prompt, which extends #26384 whenever cancelledBeforeStart(source) holds, it dispatches CONTINUE_PROMPT in "Continue where you left off" with attachments: [] instead of replaying the user's actual message and attachments.

Finally, when the new server opens an existing ACP thread, AcpAdapterV2 activateSession pops initialSessionActivationFailure and fails right away with that cached error without trying runtime.loadSession or runtime.resumeSession, even when canLoadSession or canResumeSession is true on the new runtime and live recovery works.

Rather than bolting five point fixes onto those branches, it is worth designing this at the true state of the art for crash and restart recovery. Orchestration V2 can treat shutdown and boot recovery as an explicit state machine with monotonic turn leases and a server-wide draining generation set before any session scope closes or any child gets signaled. Once draining is set, handleRuntimeTerminationAtGeneration suspends nonterminal turns and subagents instead of recording transport_error. prepareForShutdown and ProjectionStore.getRecoveryThreadIds can tag every nonterminal preparing, starting, running, and waiting run in one atomic SQLite transaction before providerSessions.shutdown begins, eliminating both the concurrency: 2 bottleneck and the narrow SQL filter. Recording a durable prompt-commit receipt per run lets continueRestartedRun replay the user's exact prompt and attachments whenever quit happened before provider commit, and reserve "Continue where you left off" for turns already committed in native history. And AcpAdapterV2.testkit.ts can fault-inject quit and crash at every run phase, covering preparing, starting pre-bind and post-bind, running with active subagents, waiting on background tasks, and multi-session load, to prove zero lost turns, zero dropped attachments, and zero orphaned subagents.

Activity

  1. juliusmarminge commented on Nov 14, 2026

    @juliusmarminge
    Member

    I tried to reproduce this on the current build and could not.

    Commit 7f81aa8 (origin/main). Ran the server standalone (node apps/server/src/bin.ts serve --mode desktop --base-dir <worktree>/.t3, T3CODE_DEV_SEED=0, fresh database, continueThreadsAfterServerUpdate at its default of on). Registered a scratch ACP agent as an acpRegistry provider instance: ACP v1, loadSession supported, each prompt streams one chunk and then holds the turn open for 120 s. Started threads with t3 thread start --provider slowAcp --model slow and quit with SIGTERM to the server PID, which is how the desktop app stops its backend. (1) One thread mid-turn: shutdown recorded the run and provider turn as cancelled (not failed/transport_error) and wrote a pending provider-runtime.continue effect. On restart the agent got session/load and a new prompt, and the continuation run was running. (2) Sent SIGTERM to the ACP child and the server at the same moment: still cancelled, continuation still prepared and resumed. (3) Nine threads mid-turn: shutdown took 1.5 s and all 9 restart-continuation effects were prepared. After restart all 9 continuation runs were running, with 11 session/load calls in the agent log. No turn, prompt or session was lost. Not exercised: an ACP protocol v2 agent with followAgentWorkState (the only path where handleRuntimeTerminationAtGeneration finalizes a turn as failed), and native subagents. Claims 4-5 (a prompt cut before the provider accepted it is resent as "Continue where you left off" with attachments: []) are the existing issue #26384, with open PRs #26861 and #32413.

    Closing as not planned. If it still happens for you, reply with the exact steps and the version you are on and it will be looked at again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions