Skip to content

fix(opencode): restarted server stops an orphaned OpenCode server started through a wrapper - #34959

Merged
juliusmarminge merged 5 commits into
mainfrom
factory/issue-22324
Nov 13, 2026
Merged

juliusmarminge merged 5 commits into
mainfrom
factory/issue-22324

Conversation

@juliusmarminge

@juliusmarminge juliusmarminge commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

When OpenCode is launched through a wrapper, the ledger can record the server process while it still carries the wrapper's command line. After it execs, the full command no longer matches, so a restarted T3 server treated the group as a recycled pid and left the orphaned OpenCode server running. Ledger entries now also store the serve arguments (optional field, so version-1 entries without it still decode). On reap, a group is stopped when pid, start time and pgid match and either the exact command line or the saved serve arguments (as the command's suffix) match; the exact-command fallback is kept for wrappers that append their own options and for legacy entries. Author's approach kept unchanged; main merged in cleanly (merge commit), and one follow-up commit removes the test helper's now-unused ready/printed signal, since the wrapper test now synchronizes through fifos. The new wrapper test fails on main (Linux and Darwin observers) and passes with the fix. The legacy-entry and appended-options tests pass on main too; they guard the compatibility paths the competing PR #22209 removes. Note for the PR body: legacy ledger entries are retained and reaped by exact command, not dropped.

Fixes #22324

Reproduction on main

Proposed real-app procedure, not executed: On Linux or macOS, configure an OpenCode provider instance's Binary path to a wrapper that forks a child, exits immediately, and has the child wait on a FIFO before exec'ing the real OpenCode executable with the original arguments. Start an OpenCode thread in T3 Code; while startup is waiting, confirm the server has written the group's ledger entry, then release the FIFO and let OpenCode finish starting. Record the OpenCode PID and port, forcibly terminate only the owning T3 Code server to bypass graceful cleanup, and restart T3 Code with the same state directory. On main, the old OpenCode process and listening port remain; with this change, restart cleanup stops that recorded group. The gated fixture makes the otherwise intermittent fork-to-exec race deliberate.

Evidence

Real T3 server and OpenCode 1.19.5 on Linux, fresh synthetic state on each side. Launch an OpenCode thread through a FIFO-gated forking wrapper; confirm the ledger records its child before exec; release the gate and observe the same PID/start time/group running OpenCode on its recorded port; crash only T3 and restart with the same state. Both runs use a 200ms spawn-return delay, disable systemd agent scopes, and pause T3 after ledger creation while OpenCode boots to prevent separate wrapper-exit handling from racing the crash. No application source or ledger edits.

Before After
f313d0a6cb 7be89fd3b0
Before: orphan survives restart After: orphan stopped and port closed
PID 58105 and port 34421 remain alive after ledger removal. PID 59825 stops and port 40429 closes; server logs identify cleanup of group 59824.
Before terminal recording After terminal recording

Driver · Complete fixture sources, original terminal recordings, transcripts and sanitized server logs. Videos are rendered from timestamped live terminal output.

Not shown:

  • Unpaused wrapper startup was not demonstrated: T3's separate wrapper-exit handler ran before this OpenCode executable finished booting, so both proof runs paused T3 after ledger creation until the deliberate crash.
  • macOS behavior was not exercised; verification ran on Linux.
  • A completed model response was not verified; this proof covers real OpenCode process startup, crash survival and restart cleanup.

Tests

vp test run src/server/OpenCodeServerLedger.test.ts (packages/provider-opencode): 16/16 pass. Same tests with main's OpenCodeServerLedger.ts: 2 failed ('records a group whose spawned wrapper already exited', linux and darwin observers), 14 passed. vp run check:changed: format, lint, knip, typecheck (10 packages), tests (2 files, 18 tests) all pass.

Not in this change

  • Runtime proof with a real OpenCode wrapper binary is not done here (left to the proof step).
  • The PR description's statement about legacy ledger entries contradicts the code (entries are kept and matched by exact command) and should be rewritten when the PR body is updated.

Who did the work

  • Written by Claude Opus 5.5, in Claude Code
  • Reviewed by GPT-6.1 Sol, in Codex, and by Claude Opus 5.5, each on its own
  • Proof captured by GPT-6.1 Sol and checked by Claude Opus 5.5

Run by the t3code factory in T3 Code.

The reaper skips a wrapper's server group when the ledger recorded the
forked child before it exec'd, because the recorded command line no
longer matches. Gate the child on a fifo so the test records it pre-exec.
Fails on main; reported in #22324 item 4.

AI-assisted; verified with the tests and tools listed above.
…rted through a wrapper

The ledger recorded the member's full command line, which for a wrapper's
forked child can be the pre-exec line, and the reaper required an exact
match, so it treated the orphan as a recycled pid and left it running.
Record the serve arguments instead and match on them both when tracking
and when reaping.

Fixes #22324

AI-assisted; verified with the tests and tools listed above.
Keep the version-1 entry format with its full command line and add the
serve arguments as an optional field. The reaper stops a recorded group
when the command line still matches exactly or, for entries that carry
them, when it still ends with the serve arguments. Entries written by an
older server still decode and are reaped as before, and a wrapper that
appends its own options is still recorded and stopped by its command line.

AI-assisted; verified with the tests and tools listed above.
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:S 10-29 changed lines (additions + deletions). labels Nov 12, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Nov 12, 2026
@macroscopeapp

macroscopeapp Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 7be89fd

Macroscope's review found this PR approvable — Orphan cleanup now recognizes wrapper-started OpenCode servers after their command line changes during exec. Existing owner, process identity, and process-group checks keep cleanup confined to recorded orphaned servers.

Notes:

  • All code in this push has already been reviewed. Approvability was decided on eligibility alone.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

All 64 scenario measurements match main.

All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded 10.5 KiB 10.5 KiB 0 B (0.0%) 12.7 KiB ✅
Codex short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn running, cold open HTTP decoded 281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅
Codex long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Codex long turn running, history page HTTP decoded 92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅
Codex long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Codex long turn finished, cold open HTTP decoded 381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, socket cold open WebSocket decoded 381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, work page HTTP decoded 186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅
Codex long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Codex idle thread resume WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell snapshot HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Codex fleet shell subscribe WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Codex fleet shell resume check rows 0 0 0 0 ✅
Codex stale shell reload HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex stale shell reload rows 27 27 0 (0.0%) 27 ✅
Codex stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Codex stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex stale todo list cold open HTTP decoded 72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅
Codex stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Codex stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅
Claude short turn resume WebSocket decoded 10.6 KiB 10.6 KiB 0 B (0.0%) 12.7 KiB ✅
Claude short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn running, cold open HTTP decoded 283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅
Claude long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Claude long turn running, history page HTTP decoded 93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅
Claude long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Claude long turn finished, cold open HTTP decoded 384.2 KiB 384.2 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, socket cold open WebSocket decoded 384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, work page HTTP decoded 188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅
Claude long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Claude idle thread resume WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell snapshot HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Claude fleet shell subscribe WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Claude fleet shell resume check rows 0 0 0 0 ✅
Claude stale shell reload HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude stale shell reload rows 27 27 0 (0.0%) 27 ✅
Claude stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Claude stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude stale todo list cold open HTTP decoded 73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅
Claude stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Claude stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅

Baseline: f313d0a · PR result: 7be89fd · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The ledger now records server arguments and uses them to match processes during tracking and orphan cleanup. Tests cover wrapper-exit timing, legacy entries without recorded arguments, appended log-level arguments, and repeated port arguments.

Changes: OpenCode server ledger

Layer / File(s) Summary
Record and identify the server process
packages/provider-opencode/src/server/OpenCodeServerLedger.ts, packages/provider-opencode/src/server/OpenCodeServerLedger.test.ts
The ledger stores server arguments and uses their command-line suffix to select a process-group member. The wrapper test uses FIFO gates to record the process group before the child continues, then waits for the ready-file write.
Match orphan processes during cleanup
packages/provider-opencode/src/server/OpenCodeServerLedger.ts, packages/provider-opencode/src/server/OpenCodeServerLedger.test.ts
Orphan cleanup accepts a matching serve-argument suffix when arguments are recorded. Entries without arguments still require an exact command match. Tests cover both cases, appended --log-level=ERROR arguments, and replacing every matching port argument.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Low

Suggested reviewers: ishaanko

Merge Risk: Minimal · up to 7be89

This change makes orphan cleanup of OpenCode server processes more reliable when a wrapper process changes its command line. No actionable merge-blocking risk was identified.

Pre-merge checks: Passed 3 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Linked Issues check Inconclusive [#22324] The ledger child-command race is addressed. The reviewed diff in OpenCodeServerLedger.ts adds an optional args field and runsServe, a suffix check on the observed command. stopOrphan acce… Provide evidence at the reviewed head that the non-ledger items are implemented: the merged commits or files, and the passing tests for pointer state, SSH zombie-process setup, timeline fixture wiring, favicon loading, and update.test.ts Wi…
Passed Passed checks: 3 passed

Check name Status Explanation
Title check Passed The title names the main change: a restarted server now stops an orphaned OpenCode server that was started through a wrapper. It follows the conventional commit form and matches the ledger race fix in…
Out of Scope Changes check Passed All changes in the reviewed diff are in packages/provider-opencode/src/server/OpenCodeServerLedger.ts and its test file. The schema field args, runsServe, the stopOrphan condition, and the recording…
Description check Passed The description covers the problem, the change, the scope link, and verification in detail. It describes the root cause (wrapper command line recorded before exec), the new optional serve-arguments fi…
Full details: Title check

Explanation

The title names the main change: a restarted server now stops an orphaned OpenCode server that was started through a wrapper. It follows the conventional commit form and matches the ledger race fix in the changeset (serve-argument suffix matching for recorded process groups). The title is 91 characters, which is longer than ideal but still clear.

Full details: Out of Scope Changes check

Explanation

All changes in the reviewed diff are in packages/provider-opencode/src/server/OpenCodeServerLedger.ts and its test file. The schema field args, runsServe, the stopOrphan condition, and the recording path all serve the ledger race in #22324. The replace-to-replaceAll change in the 'leaves a serve process on another port alone' test follows from the new args field, which stores the port in both command and args. The commit that removes the unused ready signal from the test helper supports the reproduction fixture. No unrelated change was found.

Full details: Description check

Explanation

The description covers the problem, the change, the scope link, and verification in detail. It describes the root cause (wrapper command line recorded before exec), the new optional serve-arguments field with its suffix match, and the fallback kept for legacy entries and appended options. It links the fixed issue (#22324). The Verification section lists focused test results (16/16 pass, 2 failures on main) and check:changed results. Real-app before/after evidence and a proof procedure are provided, with explicit notes on what was not shown. The description also states one inconsistency: it says legacy entries are dropped, then later says they are retained and matched by exact command. The template's own header comment asks for a conventional commit title, and the PR title is not part of the supplied context, so this cannot be checked. The description also includes an attribution section naming the models used, which the template asks for. Overall the description is complete and specific, with one internal contradiction to fix.

Full details: Linked Issues check

Explanation

[#22324] The ledger child-command race is addressed. The reviewed diff in OpenCodeServerLedger.ts adds an optional args field and runsServe, a suffix check on the observed command. stopOrphan accepts either an exact command match or a serve-argument suffix match, with PID, start time, and PGID checks kept. The test change in OpenCodeServerLedger.test.ts adds two cases for legacy entries and for wrappers that append options. The author reports a 16/16 pass, but the test run is not re-verified here. [#22324] The other requested items are not shown at the reviewed head. The author states that pointer state, SSH zombie-process setup, timeline fixture wiring, favicon loading, and the Windows update.test.ts path handling were fixed by #22523, #22427, #22534, #22529, or #22557. The recent history shown at head does not list those PRs, and no diff in this PR touches those areas. The lower-confidence orchestration failure is not addressed or explained. The item on flaky release checks is a reported impact, not a code change.

Resolution

Provide evidence at the reviewed head that the non-ledger items are implemented: the merged commits or files, and the passing tests for pointer state, SSH zombie-process setup, timeline fixture wiring, favicon loading, and update.test.ts Windows path handling. Also state whether the orchestration test failure is resolved or deliberately excluded from #22324.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between a92ac70 and 7be89fd.

Files selected for processing: 1

  • packages/provider-opencode/src/server/OpenCodeServerLedger.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.

@juliusmarminge juliusmarminge added the takeover:in-progress A takeover lead is working on this PR label Nov 13, 2026
@macroscopeapp
macroscopeapp Bot dismissed their stale review November 13, 2026 16:22

Dismissing prior approval to re-evaluate 7be89fd

@juliusmarminge juliusmarminge added takeover:ready Takeover done; auto-lands when gates pass and removed takeover:in-progress A takeover lead is working on this PR labels Nov 13, 2026
@juliusmarminge
juliusmarminge merged commit d0957bc into main Nov 13, 2026
53 checks passed
@juliusmarminge
juliusmarminge deleted the factory/issue-22324 branch November 13, 2026 16:47
@juliusmarminge juliusmarminge removed the takeover:ready Takeover done; auto-lands when gates pass label Nov 13, 2026
Pipyakas pushed a commit to Pipyakas/t3code that referenced this pull request Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S 10-29 changed lines (additions + deletions). vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Flaky browser and server tests on 4-vCPU CI runners, and a Windows-only update.test.ts failure

1 participant