Repository navigation
Startup still blocks for ~7 minutes on 0.1.10 migrations after the #35421 fix, and clients only get an empty 503 #35995
Description
What happened
After the service auto-updated, T3 Code "would not come back up": systemd showed active (running), but the server was unusable for ~7 minutes and nothing said a database upgrade was running, so the user assumed it hung.
Diagnosis
#35421 (startup migrations 82/83 taking 17 min on a 32 GB statev2.sqlite) was closed by #35486. That PR stopped requests from hanging (immediate 503), committed each migration on its own and logged "N of M" progress. Its own description says it does not make readiness faster and that the 503 has an empty body. On a build with the fix, that gap is the whole user-visible problem:
- Readiness is still blocked by every pending migration. On a 15 GB database the server answered nothing useful for 6 min 48 s. Index builds over the turn_items / nodes projection tables still finish before the server serves anything.
- The 503 carries no information.
rejectRequestsUntilServing(apps/server/src/httpResponseErrorGuard.ts) answers every HTTP request and WebSocket upgrade with503,Retry-After: 1and an empty body. The progress lines exist only inboot-service.log. Users can't tell an upgrade from a hang, and restarting throws away the migration in progress. - Each release adds another expensive startup migration. On this build migration 86 (
86_QueuedTurnInputIndex, a partial index onjson_extract(payload_json, '$.inputIntent')) took as long as 82. [Bug]: 0.1.10 startup migrations 82/83 take 17 min on a 32 GB statev2.sqlite while the server listens but answers nothing #35421 only covered 82/83, and more full-table index builds keep shipping. - The root cause keeps growing.
statev2.sqlitegrows without bound ([Bug]: ClaudeReadimages stay inline as base64 in both event and projection rows (1.4 GB, 46% of statev2.sqlite, in one week) #26591, [Bug]: Deleting a thread says it permanently clears its history, but V2 keeps every event and projection row #17845). A long-lived install hits this on every update that adds an index.
Not a duplicate of #35421: the hang is fixed. What's left is the server staying unusable for the whole migration with no user-visible progress, which #35486 left out of scope.
Steps to reproduce
- Standalone systemd service with a ~15 GB
statev2.sqlite, on0.1.8-nightly.20261108.4353. - Update to
0.1.10-nightly.20261113.4616(migrations 81–87 pending); service restarts. - Open the app or
curl -i http://127.0.0.1:3773/.well-known/t3/environment. - Empty-body
503for ~7 min; progress only inboot-service.log.
Version
Bug seen on 0.1.10-nightly.20261113.4616 (updated from 0.1.8-nightly.20261108.4353). Triage CLI: 0.1.10-nightly.20261113.4616.
Environment
Linux x64 on WSL2 (kernel 6.18.40.1-microsoft-standard-WSL2), Node v26.12.1. Standalone server as a systemd user service bound to 0.0.0.0:3773. statev2.sqlite 15.1 GB.
Evidence
`boot-service.log` (the per-migration progress added by #35486):
[05:53:28.988] Running database migrations
[05:53:28.995] 81_ProjectionMessageSearchText 1 of 7
[05:53:45.644] 82_TurnItemConversationIndex 2 of 7
[05:56:16.942] 83_CommandReadIndexes 3 of 7
[05:57:34.330] 84_LastExecutedRunIndex 4 of 7
[05:57:35.645] 85_TurnItemCounts 5 of 7
[05:57:41.971] 86_QueuedTurnInputIndex 6 of 7
[06:00:11.420] 87_ProjectionProjectsCustomTitle 7 of 7
[06:00:11.443] Migrations ran successfully
[06:00:12.811] Listening on http://0.0.0.0:3773
`GET /.well-known/t3/environment` returned `503` at 05:56 and 05:58 and `200` at 06:00:18 (`databaseMigration: 87`). Service start was 05:53:24.Related issues
- [Bug]: 0.1.10 startup migrations 82/83 take 17 min on a 32 GB statev2.sqlite while the server listens but answers nothing #35421 / fix(server): long startup migrations log progress and resume after a restart #35486: same symptom; fix removed the hang, not the blocking or missing progress.
- Desktop shows a plain loading screen during long startup migrations, and closing it rolls them back #35313: desktop loading screen during migrations; web/standalone has no equivalent.
- Desktop stays windowless for 8+ minutes after an update while startup recovery holds a write transaction open #23372: desktop windowless 8+ min during startup recovery.
- [Bug]: Claude
Readimages stay inline as base64 in both event and projection rows (1.4 GB, 46% of statev2.sqlite, in one week) #26591, [Bug]: Deleting a thread says it permanently clears its history, but V2 keeps every event and projection row #17845: unboundedstatev2.sqlitegrowth.
Suggestions: build indexes after ready or in batches (note ProjectionStore.ts pins reads with INDEXED BY on the migration-82 index, so deferring needs fallback queries); put "upgrading database, N of M" in the startup 503 body so clients can show it.
Fix applied or workaround
None. Waited it out; nothing changed on the install or database.
Filed by
Claude Code (Claude Opus 5.5), via t3 triage.
Activity
fdubayteq commentedon Nov 13, 2026
Same problem on the desktop app (macOS, bundled local server), on a build newer than the one in this report. The desktop has its own version of the visibility gap: the window sits on the white splash with the T3 logo and no text or progress the whole time. The user took it for a hang and quit once mid-migration.
Environment
9a35ab0a), updated from 0.1.9-nightly.20261110.4478statev2.sqlite: 24.4 GB (page_size 4096, 5,952,314 pages, freelist_count 2,374,046, so ~9.7 GB / 40% free and ~14.7 GB live)Migration timeline from
server.trace.ndjson(UTC, 2026-11-13)From the first migration to ready took about 12 min 20 s. Migration 88 (new since this issue was filed) is another full-table index build on
orchestration_v2_projection_turn_itemsand costs as much as 82 and 86. The per-migration commits from #35486 worked: the quit during 82 lost only 82's in-progress work, and 81 was kept.What the user saw
Ustate, ~10% CPU). Noserver-child*.logwas written, because it never crashed.Desktop-side gap (in addition to the 503 body point above): #35056 made the desktop keep waiting while migration progress lines appear in the backend output tail. In this build that information stays in the main process: the splash renderer gets nothing to display. Passing the "Upgrading database (N of M)" line the desktop already parses through to the splash would have prevented the quit. A warning when closing during a migration (suggested in #35313) would have too.
Diagnosed via
t3 triagewith Claude Code (Claude Opus 5.5).AlexChambers commentedon Nov 14, 2026
Same on the macOS desktop app, 0.1.10-nightly.20261113.4656 (updated from 20261111.4544). The window sat on the logo splash for about 5 minutes with no progress, and I assumed it was broken.
statev2.sqliteis 53 GB, but 27 GB of that is free pages (freelist_count6.68M of 13.9M pages,auto_vacuum= 0). Timings fromserver.trace.ndjson:runMigrations: 148 s in total(The per-migration spans overlap, so they add up to more than the total.) The first
ProjectionStore.readShellSnapshotRowsafter startup then took 17 to 26 s.Live data is about 30 GB:
turn_items17 GB (command output 7.4 GB, dynamic tool output 4.2 GB, spread across about 10k threads) andnodes5 GB. Settings > Storage offers "Reclaim freed database space", but nothing prompts you to use it before an update rebuilds indexes over the whole file.The same 8 migrations took about 4 s on a Linux server with a smaller database.
Snowy7 commentedon Nov 14, 2026
Same on the Windows desktop app, on a newer build than the ones above: 0.1.10-nightly.20261114.4682, auto-updated from
0.1.9-nightly.20261111.4501. The window sat on a black splash with only the T3 logo, no text or progress, for about 4 minutes. The user thought it was stuck, relaunched once mid-upgrade, and started at3 triagesession before it finished.Environment: Windows 11 Pro (10.0.26300) x64, desktop app with its bundled local server on
:3773.statev2.sqliteis 20.7 GB: page_size 4096, 5,052,748 pages, freelist_count 2,064,566, so about 8.5 GB (41%) is free pages.Migration timeline from
server.trace.ndjson(UTC, 2026-11-14):Throughout, the desktop main process polled
GET http://127.0.0.1:3773/.well-known/t3/environmentabout every 100 ms and got an empty503 (StatusCode error)each time. Nothing from that reached the renderer, so the splash showed no state. So as of build 4682, the desktop still shows nothing like "upgrading database, N of M" on any platform, and each new nightly keeps adding migrations (89–91 here).Nothing was changed on the install. Waiting it out worked.
Filed by Claude Code (Claude Opus 5.5) via
t3 triage.EtanHey commentedon Nov 15, 2026
Same on the macOS desktop app, on today's build, with one extra symptom: quitting mid-migration led to "This T3 home is unavailable" on relaunch.
Environment: T3 Code (Nightly) desktop 0.1.10-nightly.20261115.4723 (
11c425b4), auto-updated from 20261114. macOS (Darwin 27.0.0), arm64.statev2.sqliteis 19.9 GB.Timeline (UTC, from
desktop.trace.ndjsonandserver.trace.ndjson):92_QueuedTurnInputIndexJsonValidtook 67.0 s,93_ProviderTurnAttemptIndex1.9 s. The splash showed no progress, so the user quit at 06:07:46.94_TurnItemMessageOrdinalIndex(CREATE INDEX … json_extract(payload_json,'$.messageId')) started. The user force-quit around 06:08:11.bootstrap found another server owning the T3 home.awaitHomeOwnershipReleasegave up after its 10 s grace, and the app showed the "T3 home is unavailable" dialog, which points at a background service. None is installed.Migrator 94endedInterruptedafter 62.6 s, which rolled the index back and released the lock.The new part:
launcherWatch.awaitLauncherLosspollsppidevery 2 s on the JS thread. The synchronousCREATE INDEXblocks that thread, so the orphan can't notice the desktop is gone until the statement returns. It keeps the ownership lock for the whole build: about 47 s here, longer than the desktop's 10 s grace (BACKEND_QUIT_STOP_TIMEOUT+ 3 s). Then the interrupt throws the finished index away. This is a gap in the #13747 fix while a migration is running. On large databases, a misleading home-ownership dialog then follows a mid-upgrade quit.Reported via T3 triage by Claude Code (Opus 5.5).