[Bug]: Background service goes down when the disk fills up and stays down, with no low-disk warning beforehand #34413

Hi, and thank you for T3 Code. I use it every day, including from my phone through T3 Connect, and the release notes on database growth and reclaiming space have been a real help.

I want to report one thing that bit me several times: when the disk fills up, the server goes down, and on a headless machine nothing brings it back until someone is at the keyboard. I searched first and found no issue for this. #25924 (open PR) fixes the "service gives up" half and has no linked issue, so this one may give it something to point at.

What happened

My data dir (~/.t3) is on a 108 GB ext4 root disk. statev2.sqlite grew to about 6 GB, and the disk filled up several times between 2026-10-08 and 2026-11-04. Each time, writes started failing with SQLITE(13) database or disk is full. In the early October incidents the server process then exited, and the systemd user unit restarted it into the same full disk until it hit StartLimitBurst. On 2026-10-12 it stopped trying after 5 failed starts and stayed down for hours. Since I use T3 Connect, I had no access until I was back at the machine.

Two details made this harder to see coming:

• ext4 reserves 5% of the disk for root, so df showed 96 to 98% used when user processes already got ENOSPC.
• Nothing in T3 Code showed that the disk was nearly full, so the first sign was a failed turn or a missing server.

Steps to reproduce

I have not run this exact setup on a clean machine, so treat it as a recipe to try. The idea is to put the data dir on a tiny filesystem so a full disk is cheap and safe to cause.

1. Create a small RAM-backed filesystem and give it to your user, for example sudo mount -t tmpfs -o size=256m tmpfs /mnt/t3-small && sudo chown "$USER" /mnt/t3-small.
2. Start the server with its data dir there: t3 serve --base-dir /mnt/t3-small (or set T3CODE_HOME). For the service case, install the systemd user unit with Environment=T3CODE_HOME=/mnt/t3-small.
3. Open a thread and let it run so the database and logs are being written.
4. Fill the filesystem: fallocate -l 250M /mnt/t3-small/filler (adjust until df shows 0 bytes available).
5. Send another message, and watch the server log and systemctl --user status t3code.service.
6. Remove the filler and check whether the server comes back by itself.

Expected behavior

• The server keeps serving what it can (reads, thread history, a clear "disk full, writes are paused" state) instead of exiting.
• If it does have to exit, the service waits and retries when space returns, instead of using up its start limit and staying down.
• A low-disk warning appears in the UI before the disk is completely full (drafted as a separate Ideas discussion below).

Actual behavior

• Runtime 0.0.45 (early October): the server process exits with an unhandled ENOSPC raised from console.log, and the launcher reports Active child exited unexpectedly (1). Restarts then fail on the still-full disk, and the unit reaches its start limit.
• Runtime 0.1.2 and 0.1.6 (the later incidents): I did not see that crash in my log. Instead, writes fail one after another (SQLITE(13), provider event logs, cache temp files), the log file itself gets torn lines, and the server restarted on its own at least once. On startup, V2 orchestration recovery completed reported terminalizedRuns: 6, so runs had been left in progress. I could not tell from the log alone whether a hard stop or a manual restart ended that stretch.
• Today I run server 0.1.7 under the 0.1.3 service launcher. I did not let the disk fill again on 0.1.7, so I cannot say whether the early October exit still happens there.

Version or commit

Server 0.1.7, service launcher 0.1.3. The early October log lines below came from runtime 0.0.45.

Environment

Linux Mint 22.3 (kernel 7.0), ext4 root (108 GB), systemd user unit t3code.service, T3 Connect enabled, headless use from a phone. Node 26.8.2 as reported in the crash output.

Logs or stack traces

Redacted to ~. First the database error that precedes the exit (runtime 0.0.45):

WARN provider runtime ingestion failed to process event
  cause: 'PersistenceSqlError: SQL error in OrchestrationCommandReceiptRepository.upsert:query: SQLITE(13) database or disk is full
    [cause]: Error: database or disk is full'

Then the exit. The write that fails is a plain console.log:

node:events:505
    throw er; // Unhandled 'error' event
Error: ENOSPC: no space left on device, write
    at writeSync (node:fs:1211:3)
    at SyncWriteStream._write (node:internal/fs/sync_write_stream:27:5)
    at [kWriteToConsole] (node:internal/console/constructor:311:16)
    at console.log (node:internal/console/constructor:414:26)
  errno: -28,
  syscall: 'write',
  code: 'ENOSPC'
[service-launcher] Active child exited unexpectedly (1).

Restart attempts on the full disk, repeated:

ERROR effect/sql/SqlError: Failed to prepare statement
  [cause]: Error: disk I/O error
[service-launcher] Active child exited unexpectedly (1).

Later incidents (runtime 0.1.6), the server stays up and logs failures:

WARN SQLite checkpoint worker unavailable; checkpoints run on the main connection
  { reason: 'database or disk is full' }
WARN provider event log write or retention failed
  { filePath: '~/.t3/userdata/logs/provider/events.<thread>.log', errorTag: 'RotatingFileSinkError' }
ERROR PlatformError: ENOSPC: no space left on device, mkdtemp '~/.t3/caches/codex.json.<random>'

The unit file (from apps/server/src/cloud/bootService.ts) has Restart=always, RestartSec=5, StartLimitIntervalSec=300, StartLimitBurst=5, and StandardOutput=append:~/.t3/userdata/logs/boot-service.log.

My reading of the cause (a hypothesis, not confirmed)

The unit sends stdout to boot-service.log, which is on the same disk as the database. When that disk is full, the log write itself fails with ENOSPC. Node throws it as an unhandled error event on the stdout stream, so the process exits even if everything else would have survived. That would explain why the exit shows up in a console.log stack and not in a database call. I only have the 0.0.45 stack, so I cannot say whether current builds still do this.

Workaround

Free space by hand, then systemctl --user reset-failed t3code.service && systemctl --user start t3code.service. To stop the unit from giving up, I added a drop-in with StartLimitIntervalSec=0.

Related

• fix(server): pause service restart when disk is full #25924 (open PR): the launcher waits and retries while free space is low instead of burning the start limit. It matches what I saw, and I would be glad to see it land.
• fix: database errors show actionable recovery guidance #20748 (merged): clear recovery text for SQLITE_FULL in the thread error banner.
• fix(server): persist run failures after storage recovers #19646 (merged) and fix(server): V2 runs no longer look busy forever after their events stop saving #14856 (open PR): failed runs get recorded after storage recovers.
• [Bug]: Server: service self-update stops the server before checking the database backup fits #14515, fixed by fix(server): service updates no longer fail on full disks or large databases #19311: updates check free space before stopping the server. This one is about the disk filling up in normal use, outside an update.
• Threads remain stuck on “Loading messages…” after disk space is freed #10418: threads stuck on "Loading messages" after disk space was freed.
• statev2.sqlite grows without bound (33 GB in 7 weeks): retention/compaction for orchestration events and turn items #28903, fix(server): migrated databases can return freed space to disk #30066, feat: Settings reclaims freed database space in one click #32941: database growth and reclaiming space, the usual cause of the full disk.
• boot-service.log is never rotated and grows without limit #24755 and PR fix(server): rotate the boot service log in place #24765: boot-service.log is never rotated, and it sits on the same disk.

I am happy to test a nightly on the small-filesystem setup above, or to share more of the log. Thanks again for the work on this.