Skip to content

fix(server): Grok hand-off turns settle after the worktree changes - #35390

Open
juliusmarminge wants to merge 3 commits into
mainfrom
factory/issue-35274
Open

juliusmarminge wants to merge 3 commits into
mainfrom
factory/issue-35274

Conversation

@juliusmarminge

Copy link
Copy Markdown
Member

AcpAdapterV2.interruptTurn takes runtimeTransitionPermit and runs its transition in a fiber forked with Effect.forkIn(sessionScope, ...), releasing the permit via Effect.ensuring inside that fiber. After a worktree hand-off, ProviderSessionManager.detach has already closed the session scope; Effect's forkIn then returns an unstarted, already-interrupted fiber, so the ensuring/release never runs. The first interrupt exits Interrupted and the permit leaks; the ingestion-failure cleanup's second interruptTurn blocks forever on the permit, so the failed terminal is never written and the run stays running with later messages queued. The fix forks the transition with Effect.forkDetach (same startImmediately/uninterruptible options) when sessionScope.state is already Closed, so the transition actually runs and releases the permit; with an open scope it still forks into the session scope as before. No regex or parser added. The live Grok repro was not rerun; the fix is shown by the unit test.

Fixes #35274

Reproduction on main

Commit fd8e386 (origin/main). Isolated dev server: T3CODE_DEV_SEED=0 vp run dev in a fresh worktree. Enabled the grok provider instance in the worktree's settings.json, added a fresh one-commit git repo with t3 project add, ran t3 thread start --provider grok --model grok-4.6 --wait (turn 1 completed, reply 'ready'), then t3 thread send <thread> "Call the t3_worktree_handoff MCP tool now... Do not pass a continuationPrompt...". Server trace: mcp.tool.t3_worktree_handoff Success (589 ms); AcpAdapterV2.interruptTurn exit Interrupted ('All fibers interrupted without error', 16 ms) inside ProviderSessionManager.detach (Success); then WARN 'orchestration V2 provider event ingestion failed' with ProviderAdapterEventStreamError -> 'Workspace changed.'. The second AcpAdapterV2.interruptTurn span (from the ingestion-failure cleanup) never ended, and 'failed to interrupt turn after ingestion failure' was never logged. t3 thread show reported the thread running with the hand-off worktree bound; a follow-up t3 thread send returned Queued. Still running more than a minute later. Deviations: model grok-4.6 (grok-4.7-build-fast isn't on this machine's model list), and a local, uncommitted change skipping AcpSessionRuntime's 'did not advertise configured authentication method' check, because this machine's Grok 1.0.77 with XAI_API_KEY advertises only grok.com auth (it does accept xai.api_key). Failing test: AcpAdapterV2.test.ts 'settles interrupts that arrive after the session scope has closed' fails on main with 'an interrupt nobody cancelled must not exit as interrupted'. It passes with a scratch change that forks the transition detached instead of into the closed sessionScope; that change was reverted and not committed.

Evidence

Being captured: before and after in the real app. This section is filled in before the pull request can merge.

Tests

vp test run src/orchestration-v2/Adapters/AcpAdapterV2.test.ts -t "after the session scope has closed" (apps/server): passes with the fix, fails without it ('an interrupt nobody cancelled must not exit as interrupted'). vp test run src/orchestration-v2/Adapters/AcpAdapterV2.test.ts: 105/105 passed. vp run check:changed: format, lint, knip, graphql, typecheck (14 packages), tests (2 files) all pass on the second run; the first run failed only on format, fixed with vp fmt.

Not in this change

  • Other Effect.forkIn(sessionScope) call sites in packages/provider-acp/src/server/adapter.ts have the same closed-scope behaviour but do not hold the transition permit; I left them unchanged.
  • The ingestion-failure cleanup in RunExecutionService.ts calls interruptTurn without the session manager's 15 s timeout wrapper; this fix does not change that.

AI-assisted; verified with the tests and tools listed above.

Reproduces #35274: interruptTurn forks its transition into a closed
session scope, the fiber never starts, the transition permit leaks, and
the next interrupt waits forever.

AI-assisted; verified with the tests and tools listed above.
AcpAdapterV2.interruptTurn forked its transition into the session scope.
After a worktree hand-off detaches the session, that scope is closed, so
the fork never started, the transition permit leaked, and the ingestion
cleanup's interrupt waited on it forever, leaving the run running. Fork
detached when the session scope is already closed.

Fixes #35274

AI-assisted; verified with the tests and tools listed above.
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:XS 0-9 changed lines (additions + deletions). labels Nov 12, 2026
@macroscopeapp

macroscopeapp Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 73080b8

Macroscope's review found this PR approvable — The change lets interrupt cleanup finish after a session scope closes, preventing a leaked permit from leaving hand-off turns stuck. Normal open-session behavior and existing teardown safeguards remain unchanged.

No code changes detected at 973fe5f. Prior analysis still applies.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

interruptTurn now starts its transition fiber independently when the session scope is closed. A regression test covers normal and restart-requesting interrupts after scope closure.

Changes: ACP interrupt handling

Layer / File(s) Summary
Transition fiber lifecycle and regression coverage
packages/provider-acp/src/server/adapter.ts, apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.test.ts
When the session scope is closed, interruptTurn uses forkDetach; otherwise, it uses forkIn. Both paths start immediately and run uninterruptibly. The test closes the scope during a hanging prompt, then checks interrupts with and without a runtime-restart request.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: maria-rcks, stienswout, only21mil

Merge Risk: Minimal · up to 973fe

The change makes interrupts after a closed session scope settle, which should fix hand-off turns left running. No concrete merge-blocking risk was found; the live Grok reproduction was not rerun, so a quick manual check is reasonable.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the fix (Grok hand-off turns settling after the worktree changes), which matches the changeset: interruptTurn now forks the transition detached when the session scope is closed, so the…
Description check Passed The description follows the template well. It gives a problem statement with the mechanism and a change section explaining the forkDetach switch for closed scopes. It names the issue (#35274) in Scope…
Linked Issues check Passed Directly linked issue #35274 requires that a Grok t3_worktree_handoff turn settles and that the next message runs in the new worktree. The reported hang follows from a leaked runtimeTransitionPermit. …
Out of Scope Changes check Passed The whole-PR diff against the merge base contains two code changes. (1) packages/provider-acp/src/server/adapter.ts: a 7-line change in AcpAdapterV2.interruptTurn that selects forkDetach or forkIn by …
Full details: Title check

Explanation

The title names the fix (Grok hand-off turns settling after the worktree changes), which matches the changeset: interruptTurn now forks the transition detached when the session scope is closed, so the permit is released. The title is specific and follows the conventional commit form. It does not name the mechanism (closed scope, forkDetach), but it still identifies the main change.

Full details: Description check

Explanation

The description follows the template well. It gives a problem statement with the mechanism and a change section explaining the forkDetach switch for closed scopes. It names the issue (#35274) in Scope and includes a reproduction on main. Verification lists focused tests, the full file result, and the check run. The Evidence section is a placeholder that says it is still being captured and must be filled before merge. The Scope section does not explain why a fix without an issue is needed, but it links the issue. The template's commit-title guidance is followed. The Evidence placeholder is the main gap.

Full details: Linked Issues check

Explanation

Directly linked issue #35274 requires that a Grok t3_worktree_handoff turn settles and that the next message runs in the new worktree. The reported hang follows from a leaked runtimeTransitionPermit. When the session scope is closed, Effect.forkIn(sessionScope, ...) returns a fiber that never starts, so its Effect.ensuring release never runs. The second interruptTurn then waits on the permit. The whole-PR diff (merge base to head) changes only packages/provider-acp/src/server/adapter.ts, in AcpAdapterV2.interruptTurn. When sessionScope.state._tag is Closed, the transition forks with Effect.forkDetach using the same startImmediately and uninterruptible options. When the scope is open, it still forks with Effect.forkIn(sessionScope, ...). This addresses the root cause described in #35274. The regression test 'settles interrupts that arrive after the session scope has closed' (apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.test.ts) covers the closed-scope path. The PR reports that the test fails without the fix and passes with it. The live Grok reproduction was not rerun, so the end-to-end thread state after hand-off is not observed in this review. The Codex, Claude, and Muse rows in the issue are contextual only.

Full details: Out of Scope Changes check

Explanation

The whole-PR diff against the merge base contains two code changes. (1) packages/provider-acp/src/server/adapter.ts: a 7-line change in AcpAdapterV2.interruptTurn that selects forkDetach or forkIn by sessionScope state. This is the fix for #35274. (2) apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.test.ts: a regression test for interrupts after scope close, which supports that fix. Both changes connect directly to the linked issue. The PR states that other Effect.forkIn(sessionScope) call sites and the ingestion-failure timeout wrapper are deliberately unchanged, and the whole-PR adapter diff does not touch them. The Evidence section is a placeholder and does not affect code scope. The previous-head diff shows many unrelated hunks, but those reflect the earlier reviewed head and are not part of the whole-PR scope; they are not counted here.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Usage-based review receipt

Note

This review was completed with usage-based billing: files reviewed beyond your plan's included limits are billed at $0.25/file. View usage-based billing.

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 73080b8 and 973fe5f.

Files selected for processing: 2

  • apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.test.ts
  • packages/provider-acp/src/server/adapter.ts

Limit details: You’ve used all 10 included reviews currently available.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

All 64 scenario measurements match main.

All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded 10.5 KiB 10.5 KiB 0 B (0.0%) 12.7 KiB ✅
Codex short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn running, cold open HTTP decoded 281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅
Codex long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Codex long turn running, history page HTTP decoded 92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅
Codex long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Codex long turn finished, cold open HTTP decoded 381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, socket cold open WebSocket decoded 381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, work page HTTP decoded 186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅
Codex long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Codex idle thread resume WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell snapshot HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Codex fleet shell subscribe WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Codex fleet shell resume check rows 0 0 0 0 ✅
Codex stale shell reload HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex stale shell reload rows 27 27 0 (0.0%) 27 ✅
Codex stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Codex stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex stale todo list cold open HTTP decoded 72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅
Codex stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Codex stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅
Claude short turn resume WebSocket decoded 10.6 KiB 10.6 KiB 0 B (0.0%) 12.7 KiB ✅
Claude short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn running, cold open HTTP decoded 283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅
Claude long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Claude long turn running, history page HTTP decoded 93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅
Claude long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Claude long turn finished, cold open HTTP decoded 384.2 KiB 384.2 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, socket cold open WebSocket decoded 384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, work page HTTP decoded 188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅
Claude long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Claude idle thread resume WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell snapshot HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Claude fleet shell subscribe WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Claude fleet shell resume check rows 0 0 0 0 ✅
Claude stale shell reload HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude stale shell reload rows 27 27 0 (0.0%) 27 ✅
Claude stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Claude stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude stale todo list cold open HTTP decoded 73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅
Claude stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Claude stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅

Baseline: 00f617d · PR result: 973fe5f · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@juliusmarminge

Copy link
Copy Markdown
Member Author

The automated pipeline stopped on this pull request and will not merge it:

the capture step could not show the problem on the base commit, so there is nothing to prove fixed.

On head: enabled Grok, added a fresh project, started Handoff proof with 'Reply ready.' and observed completion. Sent 'Call the t3_worktree_handoff MCP tool now to move this thread to a new worktree on a new branch named handoff-proof. Do not pass a continuationPrompt. Do not run other tools.' The worktree was bound, but the turn failed with 'The provider event stream closed unexpectedly' and retained a Waiting handoff row. Sent 'In the new worktree, reply follow-up complete.'; it failed to open the provider session, while the stale Waiting row remained. On base: the initial 'Reply ready.' turn and retries failed to open the provider session because the configured authentication method was not advertised, preventing the handoff reproduction. Reviewed both recordings and the head screenshot before uploading. Stopped both servers and removed both worktrees and scratch files.

It stays open for a person to finish or close.

@juliusmarminge juliusmarminge added the takeover:in-progress A takeover lead is working on this PR label Nov 13, 2026
@juliusmarminge juliusmarminge added takeover:blocked Takeover blocked; reason in the PR body and removed takeover:in-progress A takeover lead is working on this PR labels Nov 13, 2026
@leonhoulier

Copy link
Copy Markdown

Note

🤖 Claude Opus 5.5 responding on behalf of leonhoulier

I ran the live Grok repro this PR still needs. The hang from #35274 no longer reproduces on current main, so the app can't show a before and after for this fix.

Setup: macOS, Grok CLI 1.0.77, grok-4.7-build-fast, signed in normally (the auth check that blocked the capture step didn't fire here). Two isolated dev servers ran side by side from fresh homes: main at 2dd6cddbc3, and this PR's head 973fe5f7fc merged onto that commit, which merged cleanly. Each thread learned a code word, then called t3_worktree_handoff with no continuationPrompt. A follow-up asked for pwd, the branch and the code word. These are the same steps that hung 3 of 3 times when I filed #35274.

Hand-off turn Next message
main at 65c0195b46 / e9e934bb3d (11-12) stays running, later messages queued (3 of 3) never runs
main at 2dd6cddbc3 completed (3 of 3) runs in the worktree, code word kept
this PR on 2dd6cddbc3 completed (2 of 2) same

Two other runs don't count. On main, my own edit to that checkout restarted the dev server mid-run. On this PR, Grok called the tool by the wrong name, so no hand-off happened.

Why it no longer hangs, as far as I can tell: the traces on both servers contain no AcpAdapterV2.interruptTurn span during the hand-off. That covers both the one inside detach that the PR description shows and the cleanup one. Other AcpAdapterV2 spans, such as startTurn, are recorded, so the missing span isn't a tracing gap. Without the first interrupt, the permit can't leak. #36360's new branch would also skip the second interrupt. Your base trace at fd8e386f7f still shows the detach interrupt; I haven't pinned down which change since then stopped it.

The adapter bug itself is still on main. With only this PR's new test applied to 2dd6cddbc3, settles interrupts that arrive after the session scope has closed fails ("an interrupt nobody cancelled must not exit as interrupted"). With the fix, AcpAdapterV2.test.ts passes 123 of 123. So this now reads as hardening for any other path that interrupts an ACP session after its scope closes, rather than the fix for #35274. Fixes #35274 may no longer fit; your call.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XS 0-9 changed lines (additions + deletions). takeover:blocked Takeover blocked; reason in the PR body vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Grok thread stays running forever after t3_worktree_handoff; later messages stay queued

2 participants