Skip to content

perf(server): streaming replies no longer flood the server with garbage - #961

Merged
badcuban merged 1 commit into
mainfrom
perf/server-stream-memory
Nov 16, 2026
Merged

badcuban merged 1 commit into
mainfrom
perf/server-stream-memory

Conversation

@badcuban

Copy link
Copy Markdown
Collaborator

While agents streamed, the server made a lot of short-lived garbage, and most of it came from work that grew with the size of the thread. Nothing was leaking (the heap after a forced collection stayed about 150 MB), but the garbage drove frequent full collections and big memory swings.

Causes and fixes:

  • Repeated session reports. Claude reports "requesting" on every request it makes. Each one became a stored session event (about 1,700 in one long reply), and each made the child-thread and link reactors load the whole thread. Updates that change nothing but their time are now skipped. The idle-signal check compares with the latest unstored report, and errors are always stored, so turn state and unread failures behave as before.
  • Finished agent messages loaded the whole thread to answer two questions. They now read that one message and ask whether the turn has a reply (ProjectionSnapshotQuery.getThreadMessage, hasAssistantMessageForTurn).
  • Streamed text was read back in full on every update to append a few words. It is now appended in SQL (appendStreamedText). An agent's reply no longer refreshes the thread's shell summary, which nothing in it can change.
  • Phone push alerts woke on every event. They now wake only for events that can start or end an alert. The idle wake still moves the cursor past the rest.
  • Traces: the filter decides from a cheap summary before a record is built.
  • Streaming markdown batching moved to orchestration/streamingMarkdownFlush.ts and now carries open code blocks, tables and line starts across batches. This fixes a real bug: a code fence split between two batches read as unclosed inline code and held the whole code block (up to 20 KB) until it ended. [ and * inside code no longer hold text either.

Measured on replayed sessions against main (isolated server, alternating runs):

  • Long working reply: garbage per streamed update 17.6 to 3.9 MB. Full collections in its first 48 s: 14 to 3. Average heap about 410 to 350 MB. Peak only about 8% lower.
  • Five short replies at once: 1.85 to 1.26 MB per update.
  • All ten finished replies are identical to main's, character for character.

While agents streamed, the server did work per update that grew with the
thread: Claude's per-request "requesting" status became a stored session
event each time (about 1,700 in one long reply), and each made two reactors
load the whole thread; a finished agent message loaded the whole thread too;
every update read the whole message text back to append to it; phone push
alerts woke on every event; and every traced span was built into a record
before the filter dropped most of them.

- Session updates that change nothing but their time are not stored. The
  idle-signal check compares with the latest unstored report, and errors
  are always stored, so turn state and unread failures behave as before.
- A finished agent message reads that one message and asks whether its
  turn has a reply (ProjectionSnapshotQuery.getThreadMessage,
  hasAssistantMessageForTurn).
- Streamed words are appended in SQL (appendStreamedText); an agent's reply
  no longer refreshes the thread's shell summary.
- Push alerts wake only for events that can start or end an alert.
- The trace filter decides from a cheap summary before a record is built.
- Streaming markdown batching moved to streamingMarkdownFlush.ts and now
  carries open code blocks, tables and line starts across batches. A code
  fence split between batches no longer holds a whole code block back
  until it ends, and code no longer holds text on `[` or `*`.

Measured on replayed sessions: garbage per streamed update 17.6 to 3.9 MB
on a long working reply (full collections in its first 48 s: 14 to 3), and
1.85 to 1.26 MB with five short replies. Finished replies are identical to
main's.
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
threadlines Skipped Skipped Nov 16, 2026 2:44pm UTC

Request Review

@badcuban
badcuban added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit d6d5bb0 Nov 16, 2026
24 checks passed
@badcuban
badcuban deleted the perf/server-stream-memory branch November 16, 2026 16:04

This branch was previously deployed

1 inactive deployment
Preview — 03551678 Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant