[juliusmarminge]

Shell thread-row reads filtered archivedAt, worktreePath and lineage.rootThreadId with json_extract on payload_json, so even a 2-thread worktree read scanned and parsed every thread payload; shell item counts joined runs by primary key just to read status, loading each run's full row. Fix: (1) selectShellThreadRows and listActiveShellIndex filter on the existing archived_at column, which the thread upsert always writes from the same payload. (2) Migration 089_ShellThreadReadIndexes adds expression indexes on CASE WHEN json_valid(payload_json) THEN json_extract(payload_json, '$.worktreePath' / '$.lineage.rootThreadId') END, and the worktree and family filters use that exact expression, so they SEARCH instead of SCAN. The json_valid guard keeps malformed payloads writable, which an existing ProjectionRecovery test requires. It is a CASE expression, not a partial index, because a json_valid partial index made the planner take it over threads_subagent_result_idx for the subagent-results recovery query. (3) The same migration adds a (run_id, status) runs index, and both shell item-count queries name it with INDEXED BY, since SQLite's unique-PK shortcut otherwise keeps the non-covering primary key. They now read only each run's status. (4) The terminal_failure subquery re-joined runs on presented.run_id, which is the same row as presented; it now reads presented directly, mirroring the blocking_failure subquery beside it, with the same result. This keeps its error lookup on turn_items_failed_error_idx, which the new runs index had otherwise made the planner drop. No regex or parser added. No user-visible change.

Fixes #36139

Reproduction on main

Commit 4aec277 (origin/main). Started T3CODE_DEV_SEED=0 vp run dev in a fresh worktree so the checkout's own server created and migrated an empty statev2.sqlite, then stopped it. A scratch Python script cloned the server-created thread row into 7,524 synthetic threads with ~4 KB payloads: 4,070 archived, 2,980 sharing one worktreePath, 2 on a second path. It also gave one active thread A 7,352 completed runs, each with a turn_item_counts row. Restarted the server, authenticated via POST /api/auth/browser-session with the startup pairing token, and timed GET /api/orchestration/shell (header x-t3-orchestration-protocol: 2): 0.70 s for the full snapshot and 0.39-0.45 s with omitIdleSubagents=1&omitOlderSettled=1, against 0.02 s on the empty DB. The 69 MB DB was held in memory, so absolute times are far below the reporter's 49 GB. sqlite3 on the same DB confirmed the statement shapes. The {worktreePath} filter plans as SCAN t with json_extract on every payload (13 ms for 2 matching threads). The archived filter on payload JSON took 13 ms against 1 ms on the existing archived_at column. The shell item-count query reads each run's table row through SEARCH r USING INDEX sqlite_autoindex_..._runs_1 (run_id=?), not a covering index: 6 ms for thread A, against 1 ms summing counts without the join. Two query-plan assertions added to existing tests in apps/server/src/orchestration-v2/ProjectionStore.test.ts fail on main: 'SQL shells read a subagent family and a worktree's threads' (the plan includes 'SCAN t') and 'reads shell runs and item counts without walking a thread's history' (SEARCH run USING INDEX sqlite_autoindex_orchestration_v2_projection_runs_1, not COVERING INDEX).

Evidence

Selective server-read proof, with reduced run-history scale.

Steps: create fresh databases on both commits; seed 7,524 synthetic threads through app commands, including 4,070 archived threads and a two-thread target worktree; complete 1,536 Replay turns in thread A; restart servers; execute 25 identical authenticated HTTP/RPC/MCP request rounds. Medians below use the final 20 rounds. Environment: Linux, Node development configuration, loopback transports, warm cache.

Before	After
4aec277f79	f91a7d32cf
Terminal recording · Transcript	Terminal recording · Transcript

The two-thread worktree lookup fell from 22.69 ms to 4.13 ms (5.5×), returning exactly two threads throughout. The single-root family lookup fell from 21.73 ms to 2.59 ms (8.4×). Full shell: 343.08 → 322.37 ms; trimmed shell: 384.16 → 353.76 ms; t3_thread_list: 114.89 → 103.02 ms. The broader reads improved only modestly; this does not establish removal of the reported multi-second stalls.

Seed/measurement driver · Replay-history driver · MCP authentication helper · Reproduction instructions · Raw summary

Limitations: the attempted 7,352-run seed slowed enough to exceed the projected budget, so both sides were equalized at 1,536 completed runs. Title filler also enlarges snapshot responses. Charts summarize runtime measurements; videos render live terminal casts, not UI captures.

Not shown:

• The exact 7,352-run reproduction was not completed: serial Replay fixture creation slowed to roughly 1.5 seconds per turn, projecting beyond the budget. Both sides contain exactly 1,536 completed runs instead.
• The reporter's 49 GB database, 7-second app opening and 10-second thread listing were not reproduced. Browser rendering and cold-storage performance were not measured; title filler also enlarges full snapshot responses.
• The covering run-status index's individual contribution, archive-only reads, explicit thread-id batches, multi-member subagent families and terminal failed-run lookup were not separately isolated.

Tests

Repro tests failed first on origin/factory/repro-36139 ('SQL shells read a subagent family and a worktree's threads' and 'reads shell runs and item counts without walking a thread's history'). After the fix: vp test run on ProjectionStore.test.ts, reconcileV2PreviewMigration.test.ts, Sqlite.test.ts, ProjectionRecovery.test.ts, ProjectionControlReads.test.ts, ThreadSearch.test.ts, Orchestrator.control-reads.test.ts and Migrations.test.ts: 8 files, 163 tests passed. vp run check:changed: format, lint, knip, graphql, typecheck (3 packages) and tests (4 files, 77 tests) all passed. A sqlite3 EXPLAIN QUERY PLAN check confirmed the CASE-expression family filter uses its index (SEARCH t USING INDEX (=?)), including with a malformed payload row present.

Not in this change

• The unsettledOnly filter (settledAt / settledOverride) still uses json_extract on the payload. It only narrows a set that other filters already select, so I didn't add an index for it.
• listActiveShellIndex still reads the lineage, settled, pinned and snoozed fields through json_extract for each active thread. That is inherent to building the index and was not changed.
• perf(server): active thread reads filter on the archived_at column #28992 (archivedAt column for other reads) was not touched beyond the shell reads named in this issue.

Who did the work

• Written by Claude Opus 5.5, in Claude Code
• Reviewed by GPT-6.1 Sol, in Codex, and by Claude Opus 5.5, each on its own
• Proof captured by GPT-6.1 Sol and checked by Claude Opus 5.5

Run by the t3code factory in T3 Code.

[macroscopeapp]

Approvability

Verdict: Approved at f91a7d3

Macroscope's review found this PR approvable — This optimizes shell reads using targeted SQLite indexes and existing projection columns. The migration adds indexes without rewriting customer records or changing product defaults.

^{You can add or adjust custom eligibility rules. Learn more.}

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Codex	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Codex	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Codex	Live turn WebSocket decoded	21.5 KiB	21.5 KiB	0 B (0.0%)	24.4 KiB	✅
Codex	Live turn messages	1	1	0 (0.0%)	8	✅
Claude	Total thread wire	5.0 KiB	5.0 KiB	0 B (0.0%)	6.8 KiB	✅
Claude	Thread snapshot wire	3.8 KiB	3.8 KiB	0 B (0.0%)	4.9 KiB	✅
Claude	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Claude	Live turn WebSocket decoded	21.9 KiB	21.9 KiB	0 B (0.0%)	24.4 KiB	✅
Claude	Live turn messages	1	1	0 (0.0%)	8	✅

Other scenarios

All 64 scenario measurements match main.

All 64 scenario measurements

Provider	Scenario	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	short turn resume	WebSocket decoded	10.5 KiB	10.5 KiB	0 B (0.0%)	12.7 KiB	✅
Codex	short turn resume	WebSocket messages	2	2	0 (0.0%)	2	✅
Codex	long turn running, cold open	HTTP decoded	281.7 KiB	281.7 KiB	0 B (0.0%)	302.7 KiB	✅
Codex	long turn running, cold open	rows	75	75	0 (0.0%)	75	✅
Codex	long turn running, history page	HTTP decoded	92.4 KiB	92.4 KiB	0 B (0.0%)	107.4 KiB	✅
Codex	long turn running, history page	rows	75	75	0 (0.0%)	75	✅
Codex	long turn finished, cold open	HTTP decoded	381.5 KiB	381.5 KiB	0 B (0.0%)	410.2 KiB	✅
Codex	long turn finished, cold open	rows	151	151	0 (0.0%)	151	✅
Codex	long turn finished, socket cold open	WebSocket decoded	381.6 KiB	381.6 KiB	0 B (0.0%)	410.2 KiB	✅
Codex	long turn finished, socket cold open	WebSocket messages	2	2	0 (0.0%)	2	✅
Codex	long turn finished, socket cold open	rows	151	151	0 (0.0%)	151	✅
Codex	long turn finished, work page	HTTP decoded	186.2 KiB	186.2 KiB	0 B (0.0%)	200.2 KiB	✅
Codex	long turn finished, work page	rows	150	150	0 (0.0%)	150	✅
Codex	idle thread resume	WebSocket decoded	65 B	65 B	0 B (0.0%)	200 B	✅
Codex	idle thread resume	WebSocket messages	1	1	0 (0.0%)	1	✅
Codex	fleet shell snapshot	HTTP decoded	42.4 KiB	42.4 KiB	0 B (0.0%)	53.7 KiB	✅
Codex	fleet shell snapshot	rows	27	27	0 (0.0%)	27	✅
Codex	fleet shell subscribe	WebSocket decoded	65 B	65 B	0 B (0.0%)	200 B	✅
Codex	fleet shell subscribe	WebSocket messages	1	1	0 (0.0%)	1	✅
Codex	fleet shell resume check	HTTP decoded	18 B	18 B	0 B (0.0%)	100 B	✅
Codex	fleet shell resume check	rows	0	0	0	0	✅
Codex	stale shell reload	HTTP decoded	42.4 KiB	42.4 KiB	0 B (0.0%)	53.7 KiB	✅
Codex	stale shell reload	rows	27	27	0 (0.0%)	27	✅
Codex	stale shell resume	WebSocket decoded	140 B	140 B	0 B (0.0%)	400 B	✅
Codex	stale shell resume	WebSocket messages	2	2	0 (0.0%)	2	✅
Codex	stale todo list cold open	HTTP decoded	72.4 KiB	72.4 KiB	0 B (0.0%)	83.0 KiB	✅
Codex	stale todo list cold open	rows	75	75	0 (0.0%)	75	✅
Codex	stale todo list cold open	nodes	2	2	0 (0.0%)	4	✅
Codex	stale todo list cold open	subagents	2	2	0 (0.0%)	4	✅
Codex	stale todo list cold open	plans	1	1	0 (0.0%)	2	✅
Codex	stale todo list cold open	providerTurns	1	1	0 (0.0%)	2	✅
Codex	stale todo list cold open	checkpoints	1	1	0 (0.0%)	2	✅
Claude	short turn resume	WebSocket decoded	10.6 KiB	10.6 KiB	0 B (0.0%)	12.7 KiB	✅
Claude	short turn resume	WebSocket messages	2	2	0 (0.0%)	2	✅
Claude	long turn running, cold open	HTTP decoded	283.3 KiB	283.3 KiB	0 B (0.0%)	302.7 KiB	✅
Claude	long turn running, cold open	rows	75	75	0 (0.0%)	75	✅
Claude	long turn running, history page	HTTP decoded	93.4 KiB	93.4 KiB	0 B (0.0%)	107.4 KiB	✅
Claude	long turn running, history page	rows	75	75	0 (0.0%)	75	✅
Claude	long turn finished, cold open	HTTP decoded	384.2 KiB	384.2 KiB	0 B (0.0%)	410.2 KiB	✅
Claude	long turn finished, cold open	rows	151	151	0 (0.0%)	151	✅
Claude	long turn finished, socket cold open	WebSocket decoded	384.3 KiB	384.3 KiB	0 B (0.0%)	410.2 KiB	✅
Claude	long turn finished, socket cold open	WebSocket messages	2	2	0 (0.0%)	2	✅
Claude	long turn finished, socket cold open	rows	151	151	0 (0.0%)	151	✅
Claude	long turn finished, work page	HTTP decoded	188.1 KiB	188.1 KiB	0 B (0.0%)	200.2 KiB	✅
Claude	long turn finished, work page	rows	150	150	0 (0.0%)	150	✅
Claude	idle thread resume	WebSocket decoded	66 B	66 B	0 B (0.0%)	200 B	✅
Claude	idle thread resume	WebSocket messages	1	1	0 (0.0%)	1	✅
Claude	fleet shell snapshot	HTTP decoded	43.0 KiB	43.0 KiB	0 B (0.0%)	53.7 KiB	✅
Claude	fleet shell snapshot	rows	27	27	0 (0.0%)	27	✅
Claude	fleet shell subscribe	WebSocket decoded	66 B	66 B	0 B (0.0%)	200 B	✅
Claude	fleet shell subscribe	WebSocket messages	1	1	0 (0.0%)	1	✅
Claude	fleet shell resume check	HTTP decoded	18 B	18 B	0 B (0.0%)	100 B	✅
Claude	fleet shell resume check	rows	0	0	0	0	✅
Claude	stale shell reload	HTTP decoded	43.0 KiB	43.0 KiB	0 B (0.0%)	53.7 KiB	✅
Claude	stale shell reload	rows	27	27	0 (0.0%)	27	✅
Claude	stale shell resume	WebSocket decoded	140 B	140 B	0 B (0.0%)	400 B	✅
Claude	stale shell resume	WebSocket messages	2	2	0 (0.0%)	2	✅
Claude	stale todo list cold open	HTTP decoded	73.4 KiB	73.4 KiB	0 B (0.0%)	83.0 KiB	✅
Claude	stale todo list cold open	rows	75	75	0 (0.0%)	75	✅
Claude	stale todo list cold open	nodes	2	2	0 (0.0%)	4	✅
Claude	stale todo list cold open	subagents	2	2	0 (0.0%)	4	✅
Claude	stale todo list cold open	plans	1	1	0 (0.0%)	2	✅
Claude	stale todo list cold open	providerTurns	1	1	0 (0.0%)	2	✅
Claude	stale todo list cold open	checkpoints	1	1	0 (0.0%)	2	✅

Baseline: 4aec277 · PR result: f91a7d3 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 108.5 KiB
• Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

[coderabbitai]

No actionable comments were generated in the recent review.

Walkthrough

Shell snapshot reads now use indexed archive and JSON filters, avoid a runs-table join for terminal-failure lookup, and use the run ID/status index for item counts. Migration 89 adds the associated indexes. Query-plan tests check the selected access paths.

Changes: Shell snapshot reads

Layer / File(s)	Summary
Add and register shell read indexes apps/server/src/persistence/Migrations/089_ShellThreadReadIndexes.ts, apps/server/src/persistence/Migrations.ts, apps/server/src/persistence/reconcileV2PreviewMigration.test.ts	Migration 89 adds two JSON-expression indexes on projection threads and a composite run/status index on projection runs. The migration registry and preview-upgrade test expectations include migration 89.
Update shell thread filters apps/server/src/orchestration-v2/ProjectionStore.ts, apps/server/src/orchestration-v2/ProjectionStore.test.ts	Thread queries use the archived_at column for archive filters and guard JSON extraction with json_valid. The terminal-failure lookup no longer joins the runs table. A query-plan test checks that the worktree snapshot query does not scan all thread rows.
Use indexed run status for item counts apps/server/src/orchestration-v2/ProjectionStore.ts, apps/server/src/orchestration-v2/ProjectionStore.test.ts	The item-count query uses the run ID/status index. Its query-plan test checks that run-table searches use a covering index.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Refactor · Severity of issue fixed: Medium

Suggested reviewers: t3dotgg, yash-singh1

Merge Risk: Minimal · up to f91a7

This change speeds up shell snapshot reads by using indexes and is not expected to change user-visible behavior. No actionable merge-blocking risk remains.

Pre-merge checks: 4

Check name	Status	Explanation
Title check		The title names the main change: shell reads filtered by worktree or family no longer parse every thread payload. It follows conventional commit form and matches the objectives (JSON-filter avoidance …
Linked Issues check		Linked issue #36139 has no coding acceptance threshold. It asks for faster shell reads, and it names three bottlenecks. (1) Thread-row filters read payload JSON. The PR filters archived threads on the…
Out of Scope Changes check		All changes support the shell read performance objective of #36139. ProjectionStore.ts changes the shell thread-row, active-shell, and item-count queries. The terminal_failure subquery now reads prese…
Description check		The description is thorough and follows the template's sections. Problem: shell thread-row reads filter on payload_json via json_extract, so they scan and parse every thread payload, and shell item co…

Full details: Title check

Explanation

The title names the main change: shell reads filtered by worktree or family no longer parse every thread payload. It follows conventional commit form and matches the objectives (JSON-filter avoidance via archived_at and guarded expression indexes for worktreePath and lineage.rootThreadId).

Full details: Linked Issues check

Explanation

Linked issue #36139 has no coding acceptance threshold. It asks for faster shell reads, and it names three bottlenecks. (1) Thread-row filters read payload JSON. The PR filters archived threads on the existing archived_at column in selectShellThreadRows and listActiveShellIndex. It adds expression indexes for worktreePath and lineage.rootThreadId (migration 089_ShellThreadReadIndexes.ts), and the ProjectionStore filters use the same expressions. (2) Item counts read each run's full row. The PR adds a (run_id, status) index, and both shell item-count queries use INDEXED BY on it. (3) listActiveShellIndex reads payload JSON. The PR changes only its archive filter. The issue author states "We have no fix for the other JSON filters", and the PR defers the settledAt/settledOverride filters and the remaining JSON extraction. Those deferrals match the issue's own scope. I did not run the measurements. The migration file matches the PR description. Query-plan assertions in ProjectionStore.test.ts cover the changed reads.

Full details: Out of Scope Changes check

Explanation

All changes support the shell read performance objective of #36139. ProjectionStore.ts changes the shell thread-row, active-shell, and item-count queries. The terminal_failure subquery now reads presented directly, with the same result as the prior join. Migration 089 adds the three indexes. Migrations.ts and reconcileV2PreviewMigration.test.ts register migration 89. ProjectionStore.test.ts adds query-plan and SQL-trace assertions for the changed reads. No unrelated behavior change appears in the summaries. The PR states no user-visible change.

Full details: Description check

Explanation

The description is thorough and follows the template's sections. Problem: shell thread-row reads filter on payload_json via json_extract, so they scan and parse every thread payload, and shell item counts join the runs table only to read status. Change: the reads use the existing archived_at column, migration 089 adds guarded expression indexes for worktreePath and lineage.rootThreadId and a (run_id, status) runs index, and the terminal_failure subquery reads presented directly. Scope: it cites the linked issue #36139 and states it is a performance bug fix. Verification: it lists focused test files (163 passing), the repro tests that failed before the fix, query-plan checks, and before/after timings, and it states limitations and what was not measured. Minor gaps: the Problem section does not give an explicit expected-behavior statement in the template's format, and the PR title is not shown in the provided context, so conventional-commit compliance cannot be confirmed. The description is also long and includes material beyond the template, such as attribution and factory run details, but the required information is present.

Finishing Touches

Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Recent review info

Run configuration

• Source mode: Auto
• Repository: pingdotgg/t3code
• Branch: factory/issue-36139
• File: .coderabbit.config.ts
• Commit: f91a7d3
• Layers: Repository TS config
• Review profile: CHILL
• Plan: Team
• Run ID: 2fc30910-b795-4e7d-bea8-9473cd3fe026

Commits

Reviewing files that changed from the base of the PR and between 4aec277 and f91a7d3.

Files selected for processing: 5

• apps/server/src/orchestration-v2/ProjectionStore.test.ts
• apps/server/src/orchestration-v2/ProjectionStore.ts
• apps/server/src/persistence/Migrations.ts
• apps/server/src/persistence/Migrations/089_ShellThreadReadIndexes.ts
• apps/server/src/persistence/reconcileV2PreviewMigration.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.