Skip to content

fix: aggregate live text chunks and skip partial events in runLive - #1656

Open
innoprej wants to merge 1 commit into
google:mainfrom
innoprej:fix/runlive-skip-partial-events
Open

innoprej wants to merge 1 commit into
google:mainfrom
innoprej:fix/runlive-skip-partial-events

Conversation

@innoprej

@innoprej innoprej commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

2. Or, if no issue exists, describe the change:

Problem:

In live mode, what a turn leaves in the session depends on the session service. GeminiLlmConnection turns each server message into responses without aggregating text, and marks a response partial only when its message carries turnComplete: false. Runner.runLive passes every event to appendEvent, while runAsync skips partial events. So a turn's text is stored as one model event per chunk, or, when the chunks are partial, dropped by InMemorySessionService and stored chunk by chunk by FirestoreSessionService and VertexAiSessionService. ADK Python stores it once. #1648 has the details and a reproduction.

Solution:

Follow ADK Python's live connection (gemini_llm_connection.py) and runner (runners.py):

  • In GeminiLlmConnection, only the responses that stream text are partial. The connection also collects that text in a PendingText and emits it as one non-partial response before the response that completes the turn or reports an interruption (:550-559, :600-610), before a tool call (:619-627), and when thought and non-thought text alternate or a part that is neither text nor inline data arrives. Text that arrives in the same message as the end of the turn or an interruption is emitted only in that full response, as ADK Python does.
  • The full response carries the text and, when an interruption ends the text, interrupted. No response is emitted only to report the interruption again. turnComplete and the transcriptions stay on the message's own response. That response also reports the interruption if the turn completes or the message has other content, such as audio, and it reports the interruption when no text is pending. When a message streams text and also carries a transcription, the transcription gets a non-partial response of its own, so that it is still stored.
  • Runner.runLive skips partial events when persisting, as runAsync does.
Server messages Responses on main Responses with this change
Hel and lo without turnComplete, then turnComplete: true Hel, lo, turn complete partial Hel, partial lo, Hello, turn complete
The same with turnComplete: false on the chunks partial Hel, partial lo, turn complete partial Hel, partial lo, Hello, turn complete
Hel, then lo with turnComplete: true Hel, lo with turn complete partial Hel, Hello, turn complete
Hel, then interrupted: true Hel, interrupted partial Hel, Hel with interrupted
Checking, then a tool call Checking, tool call partial Checking, Checking, tool call

runLive now hands only the non-partial events to the session service. Since every text chunk is partial, a turn that completes, is interrupted or calls a tool stores its text once, whether or not the Live API sends turnComplete: false on the intermediate messages, the question #1648 left open.

Responses that do not stream text, such as audio, transcriptions and interruptions, are no longer partial; on main they were partial whenever their message carried turnComplete: false. If the Live API sends that flag, InMemorySessionService now stores them, as the other session services already did; otherwise nothing changes for them. ADK Python does not mark audio or interruptions partial either. Four differences from ADK Python remain:

  • Live model audio is stored like any other event, as on main when the messages do not carry turnComplete: false. When they do, InMemorySessionService now stores it too, and the stored audio is sent back with the history of the next live connection. ADK Python keeps live model media with inline data out of the session in its runner (_should_append_event) and out of the history it sends (send_history). runLive passes partial events to appendEvent and never aggregates text chunks, unlike ADK Python #1648 lists this as a separate difference, and this PR adds neither check.
  • Transcriptions are not aggregated yet, so each chunk is stored as its own event, and Contents merges adjacent ones when it replays them as history. ADK Python streams the chunks as partial responses and stores one finished transcription. I can send that separately.
  • A message that ends the turn or reports an interruption and also has inline data or a transcription is arranged differently. Here the full text comes first, and the rest of the message stays on one response. ADK Python emits the inline data and transcriptions first and then the full text, and a separate response carries turnComplete, or interrupted when no text is pending.
  • In a message with text followed by another part, such as [text, executableCode], the other part's response is not partial here, so it is stored. In ADK Python that response keeps partial: true.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

./mvnw -pl core clean test with this change on top of 1f1537dc (Windows 11, Microsoft Build of OpenJDK 17.0.19): the default-test and basic executions each run 3,315 tests with no failures (30 skipped), and the other five executions pass.

Test On main With this change
GeminiLlmConnectionTest.receive_textChunks_streamedAsPartialAndEmittedInFullAtTurnComplete: FakeTransport delivers Hel without turnComplete, lo with turnComplete: false, and an empty turnComplete: true run against main's connection, it gets [text:Hel, partial text:lo, turnComplete] and fails passes: [partial text:Hel, partial text:lo, text:Hello, turnComplete]
RunnerTest.runLive_partialEvent_streamedButNotPassedToSessionService, the live counterpart of runAsync_partialEvent_streamedButNotPassedToSessionService fails: appendEvent receives [partial, final] passes: [final]
Nine GeminiLlmConnectionTest.convertToServerResponse_* tests for the rules above: text that arrives with the end of the turn, a tool call, a thought switch, inline data (not partial, and does not end the text), another part (ends the text), text with a transcription, an empty text part, and text and another part in one message ([text, executableCode], and [text, inlineData] with the end of the turn) convertToServerResponse has no PendingText parameter on main pass
Six GeminiLlmConnectionTest.convertToServerResponse_* tests for interruptions: pending text, text that arrives with the interruption, and an interruption with the end of the turn, with inline data, with a transcription, and with no text convertToServerResponse has no PendingText parameter on main pass

Three existing GeminiLlmConnectionTest tests now expect the full-text response before the message's own response: withoutInterruptedField_mapsEmptyOptional, withTurnCompleteTrue_mapsPartialFalse and withContentAndUsageMetadata_emitsMultiple. withInterruptedTrue_mapsInterruptedField now expects one non-partial response with the text and interrupted. The other convertToServerResponse tests only pass a new PendingText.

Manual End-to-End (E2E) Tests:

Ran live sessions through Runner.runLive with InMemorySessionService on the Gemini API and Vertex AI; the results are in the PR conversation. No Live model I could reach returned text in modelTurn, so the text that went through the connection was the thoughts of gemini-2.5-flash-native-audio-preview-12-2025: their chunks were streamed as partial events, and the thoughts of each finished turn were stored as one event. Model text without the thought flag was not run live. A run over an in-process GeminiLiveTransport covered it through Runner.runLive, and the connection tests cover it with the real GeminiLlmConnection and its conversion, and the Runner test uses TestLlm's live connection.

Checklist

  • I have read the CONTRIBUTING.md document.
  • My pull request contains a single commit.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

  • Callers of runLive now receive a turn's text twice, as partial chunks and then as one non-partial event, as ADK Python's callers do. A caller that shows the text of every model event needs to skip the partial events or the full one.
  • When an interruption ends pending text, interrupted is on the event with the full text. A caller that looks for it only on events without content needs to read it there as well.
  • A part that is not text and shares a response with streaming text, such as [executableCode, text] or [text, inlineData] in a message that does not end the turn or report an interruption, goes out on the partial response and is not stored, as in ADK Python. On main, such a message was stored unless it set turnComplete: false.
  • If the connection closes or fails before the turn ends, the pending text is not emitted, as in ADK Python. On main, its chunks were stored when they were not partial.
  • FirestoreSessionService.appendEvent still has no partial check of its own, the smaller alignment mentioned in runLive passes partial events to appendEvent and never aggregates text chunks, unlike ADK Python #1648. With this change, neither runAsync nor runLive passes it partial events.

@hemasekhar-p

Copy link
Copy Markdown
Contributor

Hi @innoprej, Thank you for your contribution and for taking the time to submit this pull request. Our team is currently reviewing your changes, and we will reach out if we need any further information.

@kvmilos kvmilos left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rebase onto main and move your message list into the reworked FakeTransport so CI can run. GeminiLlmConnectionTest now has LiveConnectionFixture, and FakeTransport was updated to record sends.

Before we merge, could you run a live session through Runner.runLive against a Live model and post what ends up in the session? I'd like to verify that it streams the partial chunks and stores one full-text event per turn. If you can't run it, let us know and we will.

Could you change Closes: #1648 to Part of #1648, or open a follow-up issue for transcription aggregation? #1648 also covers aggregating transcription chunks, which this PR leaves for later, and we don't want that part lost when the issue closes.

if (streamedParts.size() == parts.size()) {
serverContent.modelTurn().ifPresent(builder::content);
} else if (!streamedParts.isEmpty()) {
builder.content(serverContent.modelTurn().get().toBuilder().parts(streamedParts).build());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test where a single message has both text and a non-text part, such as [text, executableCode]. No test currently hits this else if branch where only part of the message's parts are kept.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added convertToServerResponse_textAndOtherPartInOneMessage_keepsOtherPartAfterFullText for [text, executableCode]: the text comes out in full, then a response with only the executableCode part and the message's role. convertToServerResponse_textAndInlineDataWithTurnComplete_keepsInlineData covers the same branch when the turn ends, with [text, inlineData]. With the else if disabled, these two tests fail and the other tests in the class pass.

boolean turnComplete = serverContent.turnComplete().orElse(false);
if (turnComplete || serverContent.interrupted().orElse(false)) {
streamedParts.removeIf(GeminiLlmConnection::hasText);
// Unlike in ADK Python, the full text does not carry interrupted; the response below does.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you match ADK Python here by putting interrupted on the full-text response and dropping the separate marker? I saw the PR description notes you kept interrupted on the message's own response where callers read it today, but matching Python keeps the emitted events consistent across ports.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. When text is pending, the full-text response now carries interrupted, and no response is emitted only to report it again. If the turn also completes or the message has other content such as audio, that response reports it too, as in ADK Python. A transcription that follows the full text gets a response without the flag. With no pending text, the message's own response reports it as before; ADK Python sends a separate response when that message also has audio or a transcription. The remaining differences are listed in the PR description.

In this repo, BaseLlmFlow, SessionJsonConverter, the tokt codecs and the dev UI read the field and none of them relies on a separate event. For external callers, who may have looked for it only on an event without content, I added a note to the PR description.


private static LlmResponse createServerContentResponse(LiveServerContent serverContent) {
/**
* Emits the responses for a server content message. As in ADK Python, text parts are streamed in

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: please drop the "As in ADK Python" / "Unlike in ADK Python" phrasing from the code and test comments and just state the rule directly. The parity notes in the PR description are enough.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The code and test comments state the rules directly now, and the parity notes are only in the PR description.

@kvmilos kvmilos added needs update waiting on reporter Waiting for reaction by reporter. Failing that, maintainers will eventually closed it as stale. and removed needs update labels Nov 2, 2026
GeminiLlmConnection turned each live server message into responses
without aggregating text, and marked a response partial only when its
message carried turnComplete: false. Runner.runLive passed every event
to appendEvent, while runAsync skips partial events. A live turn's text
was therefore stored as one model event per chunk, or, with partial
chunks, dropped by InMemorySessionService and stored chunk by chunk by
FirestoreSessionService and VertexAiSessionService.

Follow ADK Python's live connection and runner:

- Only responses that stream text are partial. The connection also
  collects the text and emits it as one non-partial response before
  the end of the turn, an interruption or a tool call, and when thought
  and other text alternate or a part that is neither text nor inline
  data arrives. Text that arrives with the end of the turn or an
  interruption is emitted only in full.
- The full response reports an interruption that ends the text, and no
  other response is emitted just to report it.
- runLive skips partial events when persisting, as runAsync does.

Responses without text are no longer partial when their message
carries turnComplete: false. Transcriptions are not aggregated yet.
@innoprej
innoprej force-pushed the fix/runlive-skip-partial-events branch from ca73bf4 to 9435377 Compare November 5, 2026 03:25
@innoprej

innoprej commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Updated:

  • Rebased onto 1f1537dc (head 94353776). The only conflict was in GeminiLlmConnectionTest, and the message list now lives in the reworked FakeTransport.
  • Changed Closes: #1648 to Part of #1648, so the issue stays open for transcription aggregation.
  • The three inline comments are answered in their threads.

I also ran Runner.runLive against real Live models on this head, with InMemorySessionService, on both the Gemini API and Vertex AI.

I couldn't get model text in modelTurn from any Live model I could reach:

  • Gemini API: the docs say native audio models only support AUDIO, and the other models this API key lists didn't return modelTurn text either. gemini-2.5-flash-native-audio-preview-12-2025 and gemini-3.8-live reject TEXT with close code 1007.
  • Vertex AI: the model page lists text output and the Live text sample uses TEXT, but in us-central1, gemini-live-2.5-flash-native-audio closes with Text output is not supported for native audio output model. and gemini-3.8-live with Unsupported modality for Vertex Live API. Both are close code 1007. With AUDIO, both answered with audio and output transcription only, with no text or thought parts.
  • gemini-3.5-transcribe-live on Vertex AI (global) accepts TEXT, but sends speech text in inputTranscription. I didn't run gemini-3.5-live-translate-preview there, which is documented as speech-to-speech.

With AUDIO on the Gemini API, gemini-2.5-flash-native-audio-preview-12-2025 sends its thoughts as text parts, and those go through the same code. This is one session with that model: a first prompt answered in full, then a second prompt that I interrupted with speech, and the turns the model started for that speech. Audio and transcription events are counted rather than listed.

Turn Streamed Stored in the session
1 1 partial thought (396 chars), 299 audio and 25 output transcription events, the full thought (396 chars), turnComplete, usage the same without the partial event
2, interrupted 12 partial thoughts (6,917 chars in total), 242 audio and 22 output transcription events, the full thought (6,917 chars) with interrupted: true, turnComplete, usage the same without the 12 partial events
3, interrupted 2 input transcription events, 1 partial thought (240 chars), the full thought (240 chars) with interrupted: true, turnComplete, usage the same without the partial event
4, not finished 2 input transcription events, 1 partial thought (285 chars), 105 audio and 8 output transcription events the same without the partial event, and no full thought

In this session, no event carried only the interruption. The session ended with 714 stored events and no partial ones, while 15 partial events were streamed. Turn 4 has no full thought because I closed the connection before it ended, which is the pending-text case noted in the PR description.

On Vertex AI, in one gemini-live-2.5-flash-native-audio session with AUDIO that I interrupted with speech, no text or thought parts were streamed, and interrupted: true came on its own event 3 times. All 146 streamed events were stored, none of them partial.

What this doesn't show:

  • Model text without the thought flag didn't run live. In the connection, it differs from a thought only by that flag, but Contents and outputKey skip thoughts, so the live run didn't reach those steps with text. If you know a Live model that returns text in modelTurn, let me know and I'll run it.
  • Of the 714 stored events, 646 are audio chunks and 55 are output transcription chunks, each stored on its own. Both are differences listed in the PR description, not something this PR aggregates.

To cover plain text without a model, I ran the same path with an in-process GeminiLiveTransport passed through Gemini.connectLiveTransport:

  • Streamed: partial Hel, partial lo, Hello, turnComplete, partial Che, partial ck, Check with interrupted: true, turnComplete.
  • Stored: Hello, turnComplete, Check with interrupted: true, turnComplete.
  • outputKey ended as Check, and a new connection on that session got Hello and Check as model history. Content sent through LiveRequestQueue isn't stored, which this PR doesn't touch.

Two other observations:

  • In a separate short 2.5 session, 171 server messages had serverContent: 2 set turnComplete: true, 169 left it out, and none set it to false. That is one model and one run, and the text handling here doesn't depend on it.
  • gemini-3.8-live with AUDIO sent no thoughts or partial events in a two-turn session, so it doesn't exercise the aggregation. Of its 64 stored events, 31 are Unknown server message. error events: it sends messages with no field the client exposes, and convertToServerResponse has the same path for those on main. I left that out of this PR. On Vertex AI, the same model sent none of these in a two-turn session.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting on reporter Waiting for reaction by reporter. Failing that, maintainers will eventually closed it as stale.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants