Repository navigation
perf(server): streaming replies no longer flood the server with garbage - #961
Merged
Merged
Conversation
While agents streamed, the server did work per update that grew with the thread: Claude's per-request "requesting" status became a stored session event each time (about 1,700 in one long reply), and each made two reactors load the whole thread; a finished agent message loaded the whole thread too; every update read the whole message text back to append to it; phone push alerts woke on every event; and every traced span was built into a record before the filter dropped most of them. - Session updates that change nothing but their time are not stored. The idle-signal check compares with the latest unstored report, and errors are always stored, so turn state and unread failures behave as before. - A finished agent message reads that one message and asks whether its turn has a reply (ProjectionSnapshotQuery.getThreadMessage, hasAssistantMessageForTurn). - Streamed words are appended in SQL (appendStreamedText); an agent's reply no longer refreshes the thread's shell summary. - Push alerts wake only for events that can start or end an alert. - The trace filter decides from a cheap summary before a record is built. - Streaming markdown batching moved to streamingMarkdownFlush.ts and now carries open code blocks, tables and line starts across batches. A code fence split between batches no longer holds a whole code block back until it ends, and code no longer holds text on `[` or `*`. Measured on replayed sessions: garbage per streamed update 17.6 to 3.9 MB on a long working reply (full collections in its first 48 s: 14 to 3), and 1.85 to 1.26 MB with five short replies. Finished replies are identical to main's.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While agents streamed, the server made a lot of short-lived garbage, and most of it came from work that grew with the size of the thread. Nothing was leaking (the heap after a forced collection stayed about 150 MB), but the garbage drove frequent full collections and big memory swings.
Causes and fixes:
ProjectionSnapshotQuery.getThreadMessage,hasAssistantMessageForTurn).appendStreamedText). An agent's reply no longer refreshes the thread's shell summary, which nothing in it can change.orchestration/streamingMarkdownFlush.tsand now carries open code blocks, tables and line starts across batches. This fixes a real bug: a code fence split between two batches read as unclosed inline code and held the whole code block (up to 20 KB) until it ended.[and*inside code no longer hold text either.Measured on replayed sessions against main (isolated server, alternating runs):