SQLite writes run on the server's event loop, so one slow transaction disconnects every client

What happens

The server's SQLite writer connection runs node:sqlite DatabaseSync on the main thread. Reads and WAL checkpoints already run in workers (#21799, #24194), but every write transaction, and every read inside one, still blocks the event loop. On a large statev2.sqlite, a single slow transaction freezes every HTTP, WebSocket, and T3 Connect relay request until it finishes. Clients then show " is reconnecting" and "T3 Connect returned an error", and the reconnect requests queue behind the same stall.

Evidence (macOS, Mac mini M-series, 48 GB, 0.1.2 and 0.1.6)

event-loop-stalls.ndjson recorded stalls of 7–40 s while the CPU was about 60% idle overall.
server.trace.ndjson showed sql.transaction spans up to 126 s and orchestrationV2.EventSink.write up to 138 s. A plain PRAGMA auto_vacuum took 115 s in the server, and 0.05 s from the sqlite3 CLI on the same file at the same time.
/api/auth/session (EnvironmentAuth.getSessionState) took 8–31 s and was interrupted, so the relay dropped the environment.
It got worse whenever agents in T3 ran CPU-heavy tools, such as several headless Chromium captures. That load lengthens every synchronous write, and it all lands on the one thread.
Compacting the database (14.4 GB → 5.2 GB) and updating to 0.1.6 reduced the stalls but did not remove them. The underlying cause is write work running on the event loop.

Related: #23482, #23372, #21539, #24048, #25203.

Expected

Slow writes delay only the requests that need them. Auth, health, and relay traffic stay responsive.

Same symptom on Linux (WSL2), server 0.1.7-nightly.20261104.4151 (t3 serve), client = macOS desktop app connected over T3 Connect.

Evidence (2026-11-04, UTC)

No server restart (process up since 02:27Z). Client showed " is reconnecting · T3 Connect returned an error" around 07:26–07:29Z.
event-loop-stalls.ndjson: 95 stalls that day, worst 34.8 s. A 12.4 s stall at 07:25:18 was followed by cloudflared failed to accept QUIC stream: timeout: no recent network activity at 07:25:56.
server.trace.ndjson (07:20–07:35Z): 491 SQLite spans > 5 s; several sql.transaction spans ~265 s (Success). PullRequestSyncReactor.sweep up to ~437 s.
Relay log: ~98 Incoming request ended abruptly: context canceled on /api/t3-connect/mint-credential, /oauth/token, /api/auth/session, /api/orchestration/shell.
Host was under heavy load at the time (load avg ~28, kernel memory-pressure warnings) from many parallel agent processes. That matches the "worse when agents run CPU-heavy tools" note above, so it reproduces on Linux and on the latest nightly too.
This build already contains perf(server): PR sync stops reading pull requests closed over a day ago #30071, yet PR sync sweeps still ran ~7 min each under this load. fix(server): SQLite writes no longer block the event loop #30067 looks like the fix that matters here.

Contributing factor on our side: the Windows drive holding the WSL2 disk image had only ~2 GB free, which made disk I/O very slow and stretched those transactions. After moving the image to a drive with free space (10:04Z), the server recovered. Still, a slow disk should not freeze every connected client, which is what #30067 addresses.

Evidence gathered by Claude Opus 5.5 + Sonnet 5.5 (Claude Code) via the T3 triage playbook.

Note

🤖 GPT-6 Astra responding on behalf of Elliot

Adding sanitized Windows evidence and workload attribution. This supports the responsiveness concern, but does not establish synchronous SQLite writes as the sole cause in this installation.

Environment/workload: Windows 11 x64, desktop app and remote desktop server both 0.1.7. A few agent threads monitor roughly 17–20 PRs (user report). Server has 236 thread records; approximately 32 GiB RAM free, 86 GiB disk free, normal process priority. No database modifications or antivirus exclusions were made.

Observed on Nov 6, UTC:

Client thread visit at 22:56:21 waited 107.063 seconds, ending with SocketReadError at 22:58:08; asset requests failed simultaneously.
Server event-loop stalls include 5.040 seconds at 22:55:31 and 5.333 seconds at 22:58:13, with repeated 2–7 second stalls around them.
HTTP requests issued on the server host to its own 127.0.0.1:3773/.well-known/t3/environment took 12.747, 7.663, and 7.489 seconds. A later series included two 15-second timeouts. This reproduces slowness without the remote network/relay.
Reader connection acquisition waits reached 27.46 seconds. These are queueing measurements, not SQL execution measurements.
An external read-only SELECT 1 took 1.64 ms. This does not rule out slow writes, complex queries, or contention inside T3.

Workload attribution, captured process operations starting 22:44:05–23:06:11:
Deduplicated spans by (traceId, spanId), then excluded processRunner.runProcessCore when its captured ancestry already contained VcsProcess.spawn. Classified using captured parent chains:

Ancestor	Process-operation spans
ThreadPullRequestServiceV2.synchronize	271
CheckpointCaptureServiceV2.execute	168
ProviderTurnStartService.start	81
ThreadSettlementServiceV2.sweep	59
PullRequestSyncReactor.syncGroup	23
Other/unattributed	116

These are observed process-operation spans, not an OS process-start census; rotated logs can omit ancestors and these paths need not all be Git. Parent/child durations must not be summed as CPU time. Agent-shell commands are not comprehensively represented by these server spans.

Independent Node 26.13 probes on the same Windows host measured synchronous child_process.spawn return times of 134–550 ms for Git on PATH. Explicit real Git binary probes ranged 23–653 ms; even cmd.exe ranged 12–405 ms. This establishes a separate local process-creation bottleneck that can amplify T3's background work; it is not a measurement inside the live Electron backend. McAfee was active, but we have not causally attributed the delays to it.

Source audit at v0.1.7 (322f654): branch/PR discovery groups by project/worktree/branch and uses concurrency 32, with a minute-spaced sweep plus event-triggered refreshes. Linked-PR sync separately batches at concurrency 25. Existing caches and quiet-PR polling mean this is not evidence of an absence of caching. In particular, the terminal-PR path already has a 15-minute confirmation cache; #24098's original mechanism should not be assumed unchanged.

For further performance work, this workload looks useful for testing responsiveness under expensive Windows process creation while branch discovery, checkpoints, and PR metadata synchronization run together. Candidate improvements to investigate are deduplicating concurrent repository discovery across groups, coalescing refreshes around run completion/checkpoint events, bounding host-wide background process starts, and prioritizing health/interactive work. These are proposals, not reproduced defects.

We have not yet tested #30067 or the newer nightly on this installation. No raw logs, credentials, private repository names, thread content, or user home paths are included above.

Note

🤖 GPT-6.1 Sol responding on behalf of Elliot

Follow-up measurement after the user updated both desktop client and remote desktop server to 0.1.8-nightly.20261106.4288:

Updated remote backend started 2026-11-06 23:48:24.896Z.
Eight localhost environment-discovery requests at approximately 23:54Z took 480, 60, 148, 87, 59, 9, 40, and 89 ms (median 73.5 ms). Earlier 0.1.7 probes included 7–13-second responses and 15-second timeouts.
Stalls still occurred after the updated backend started: 3.614 seconds at 23:51:27Z and 2.137 seconds at 23:53:57Z. Older 5–15-second records in the same log preceded this backend and should not be attributed to this updated process.

This is a promising short sample, not proof of a fix. Both client and server changed and restarted together; we have no restart-only control and have not established equivalent background workload. We cannot attribute the improvement to a particular change. Further observation under the user's normal PR-monitoring workload is needed.

Note

🤖 GPT-6.1 Sol responding on behalf of Elliot

Approximately four-hour follow-up on 0.1.8-nightly.20261106.4288 (checked 2026-11-07 around 03:47Z): the remote backend is still the same process started at 2026-11-06 23:48:24Z.

The server's retained event-loop log contains 13 post-start stall records above its 2-second reporting threshold. Three exceed 5 seconds; maximum 12.791 seconds. No recorded post-start stall exceeds 15 seconds. This monitor samples and can miss stalls; these are recorded counts, not an exhaustive census.
Residual freezes include 7.766 seconds at 02:46:11Z and 5.627 seconds at 03:19:19Z.
Retained client traces contain three remote-environment transport-loss events at 00:58:03Z, 01:22:08Z, and 03:11:20Z. The generic detail is "disconnected"; we have not established that these are server-stall-induced or ruled out sleep/network/client activity. Rotated logs limit completeness.
Twelve current HTTP requests from the remote host to its own environment-discovery endpoint all succeeded: 1703, 699, 1751, 234, 6, 28, 18, 18, 229, 51, 1330, 1122 ms. Median 231.5 ms, maximum 1.751 seconds.
The current captured server trace file includes ongoing provider commit batches and Git operations, so the server is doing background work, but equivalence to the earlier PR-monitoring workload remains unverified.

The initial responsiveness improvement has persisted in this later sample, but stalls and disconnects remain. This is partial improvement, not resolution, and the combined update/restart experiment still cannot identify which change helped. Sanitized observations only; no private repository identifiers, connection URLs, tokens, or raw logs are included.

Note

🤖 GPT-6.1 Sol responding on behalf of Elliot

Live recurrence captured around 2026-11-07 03:59–04:04Z on the same 0.1.8 nightly. This adds a substantial host-contention confound to our earlier evidence; it should not be read as proof of a SQLite-only defect.

Client missed the remote health check at approximately 03:59:48Z. Relay-side websocket-ticket and credential requests were cancelled around 03:59:55–04:00:05Z.
Four server-host localhost environment-discovery requests took 4676, 2081, 3652 and 1732 ms.
Host-wide CPU reported 100% during this recurrence, with about 28–30 GiB free RAM.
To avoid process-instance reuse artifacts in Windows formatted counters, we sampled Get-Process CPU deltas over three seconds and matched PID plus process start time. McAfee Framework Host alone used 8.75 CPU-core equivalents in one sample. A subsequent grouped sample showed McAfee 6.26 cores, headless Chromium 3.21, Node 1.42, and the five T3 application processes together 0.36. These are short samples, not whole-session averages.
A process-tree snapshot contained 71 headless-browser processes in 17 browser roots. Most roots belonged to two active Playwright worker groups started around 03:55–03:56Z; one belonged to a separate agent-launched tool chain. Another snapshot showed 61 browser processes, so this is changing active work, not evidence that every browser is orphaned. No explicit worker-limit argument appeared in the two Playwright CLI command lines; config/environment limits were not inspected.
The current two-file server trace capture contains 62 VcsProcess.spawn spans (median 2413 ms, max 14837 ms), plus SQLite spans up to 17863 ms. Queueing, scheduling and execution are not distinguished by these elapsed maxima.

Next causal test is to compare responsiveness with bounded browser-test concurrency while preserving PR monitoring, then separately assess McAfee activity. No test processes were killed, no antivirus controls were disabled, and no configuration/database changes were made during this capture. The McAfee/test interaction and the precise cause of T3's remaining main-thread stalls are not yet proven.

Additional natural comparison a few minutes later: the two sampled Playwright runner PIDs had exited without intervention. In a new three-second CPU sample, McAfee dropped to 0.71 core equivalents. Five localhost environment-discovery requests took 29, 2, 2, 2, 1 ms. No T3 restart, antivirus change, or configuration change occurred between the degraded and recovered measurements. This strengthens the workload/contention association, but is not a randomized intervention and does not prove what drove the antivirus work.

One test configuration inspected locally had fullyParallel:true, seven viewport projects, and no explicit workers setting. Its script is simply playwright test. A capped-worker comparison is a concrete next experiment; several independent suites can otherwise multiply host-level concurrency.

Additional local triage evidence from T3 Code (Beta) desktop and local server 0.1.8 on macOS arm64.

The user reports that the whole Mac appeared unresponsive while using a Codex thread. After trying various keys, responsiveness returned with a main-process error dialog. The exact incident time and a deterministic reproduction remain unconfirmed.

The server trace contains a slow writer transaction:

COMMIT: 21,844.940 ms
sql.transaction: 21,848.296 ms
orchestrationV2.EventSink.commitCommand: 21,848.859 ms
A nearby event-loop sample: delayMaxMs=21079, cpuUserMs=609, cpuSystemMs=38, majorPageFaults=0, rssMb=309.

T3 connection/semaphore waits were 0.088708/0.012 ms. These do not measure SQLite-internal locking or operating-system I/O waits. The underlying reason for the slow commit is unknown, and this server stall has not been conclusively tied to the reported whole-Mac unresponsiveness.

The later dialog matches the stack signature in #33614 and may be a separate, secondary input/shutdown error. No workaround or local patch was applied.

Investigated by Codex (GPT-6) via t3 triage.

Another data point, from a small database on a low-memory machine: Linux desktop app 0.1.2, 11 GB RAM, SATA SSD. statev2.sqlite is 2.0 GiB, with 0 free pages and no fragmentation.

The snapshot query SELECT MAX(sequence) … WHERE stream_id = ? takes 6 ms from the sqlite3 CLI but took 18 s in the server. Writer-connection spans reached 14–24 s and reader spans up to 122 s. These line up with 10–19 s entries in event-loop-stalls.ndjson. Most of those spans had lock waits under 0.1 ms, so they were waiting on disk, not on a lock.
The trigger is memory pressure, not database size. Over 20 hours of uptime the kernel refaulted about 64 GB of evicted file pages (workingset_refault_file). When I checked, only 1.2 of the 2.0 GB were in page cache. Each cache miss becomes a disk read, and on the writer that read happens on the event loop.
Clients then show "reconnecting", as in the report above.

So a 2 GB database is enough to freeze the server once its pages fall out of cache. Compaction alone won't fix that; moving writes off the event loop (#30067) does. I'm updating to 0.1.9 and will follow up if it still happens there.

Another data point: Linux arm64 (Rockchip RK3588 SBC, 8 cores, 7.7 GB RAM, eMMC storage), server 0.1.7 run with t3 up --connect as a systemd user service. The clients are the web app (app.t3.codes), the macOS desktop app and the iOS app, all connecting over T3 Connect.

Symptom: many times a day the web and desktop clients show "rock-5c is reconnecting · T3 Connect returned an error" or get stuck syncing, and recover on their own after 10–15 s. The server process never restarted (it has been up since 2026-11-09, NRestarts=0).

One drop captured as it happened (2026-11-12, UTC):

09:34:25: ThreadSettlementServiceV2.sweep starts and runs for 34.5 s. Its spans include 77 branchPullRequest lookups and 386 GitManager.rememberPrLookupGitState calls.
09:34:25–09:35:00: writer sql.transaction / COMMIT spans of 2.0–9.3 s back to back. Single statements took 5.7 s (SELECT payload_json FROM orchestration_v2_projection_nodes WHERE node_id = ?) and 7.3 s (a thread shell query).
09:34:40: event-loop-stalls.ndjson records a 10.5 s stall at utilization 1.0 with only ~0.9 s of CPU in the 30 s window. The loop was blocked waiting, not computing.
09:34:33–09:34:59: cloudflared logs Incoming request ended abruptly: context canceled for /api/t3-connect/mint-credential and /api/orchestration/shell about every 15 s, which matches the client's reconnect and probe timeouts.
An earlier drop at 09:32–09:33 followed the same pattern: a 39.8 s sweep, a 3.7 s stall at utilization 1.0, and mint-credential cancellations.

Background numbers:

event-loop-stalls.ndjson: 117 stalls of 2 s or more between 2026-11-10 and 2026-11-12. Median 4.3 s, max 24.7 s.
SQLite span durations in the current trace file: p50 1.3 ms, p99 1.8 s, max 21.9 s.
Settlement sweeps in the retained traces (~1 h): 660 runs, median 12 ms, but 33 took over 5 s, up to 40 s.
statev2.sqlite is 812 MB with a 25 MB WAL, on eMMC.
Host IO pressure (/proc/pressure/io full) averaged about 40% over 60 s and 33% over 300 s, while CPU sat about 93% idle. Agent sessions on the same host were running docker builds and e2e tests on the same disk, and load average was about 9.

This fits the description in this issue: disk-bound writer work on the main thread blocks relay and auth traffic. Slow storage plus IO from agent work on the same disk makes it trigger with a much smaller database than the earlier reports. The settlement sweep (every minute) seems to be a regular source of writer transactions that happen to overlap these stalls.

_{Triage done with Claude Code (Claude Opus 5.5) via t3 triage.}

Note

Written by Claude Opus 5.5 (high effort) in Claude Code, on behalf of tnfssc.

Same problem on another machine, this time on Linux. I filed it separately as #35727 before finding this issue, and closed that one as a duplicate.

Machine: 24 cores, NVMe drive, btrfs, many agents running. The database was 3.6 GB. Version 0.1.10-nightly.20261112.4580.
How bad: the server froze 3 times a day at first, and 146 times on 2026-11-12. 29 of those freezes lasted over 15 seconds, long enough to disconnect clients. The longest was 243 seconds.
Cause: the server's traces show single database saves taking 9.8, 9.6 and 5.5 seconds, and nothing else on the server ran while the 9.8-second one was going.
How to reproduce it without a slow disk: run vp run dev under strace so that every 150th database write takes 3 extra seconds (-e inject=pwrite64:delay_enter=3000000), then run a few Replay conversations. On main, 49 of 163 simple requests took over a second, and the slowest took 6 seconds.

#35728 is a fix along the same lines as #30067. The comment on #35728 explains how the two differ.

Fixed by #35728, now on main: File-backed SQLite writes and transactions now run on a worker thread, so reproducing a slow database write should leave HTTP, WebSocket heartbeats and T3 Connect traffic responsive.

Navigation Menu

Repository navigation

SQLite writes run on the server's event loop, so one slow transaction disconnects every client #30065

What happens

Evidence (macOS, Mac mini M-series, 48 GB, 0.1.2 and 0.1.6)

Expected

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SQLite writes run on the server's event loop, so one slow transaction disconnects every client #30065

Description

What happens

Evidence (macOS, Mac mini M-series, 48 GB, 0.1.2 and 0.1.6)

Expected

Activity

Furizou commented on Nov 4, 2026

ElliotDrel commented on Nov 6, 2026

ElliotDrel commented on Nov 6, 2026

ElliotDrel commented on Nov 7, 2026

ElliotDrel commented on Nov 7, 2026

softmaxe commented on Nov 10, 2026

Amer-Alic commented on Nov 11, 2026

felix920506 commented on Nov 12, 2026

tnfssc commented on Nov 13, 2026

juliusmarminge commented on Nov 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions