Repository navigation
Conversation
Gemini Live occasionally answers an audio-modality turn with a text part and no audio. The plugin resolved every message as audio up front from the session config, so the agent forwarded an empty audio stream and the framework's text-only fallback (TTS, or an error log) never engaged. Resolve the modality from what arrives: audio on the first inline frame, text when the turn completes without audio while carrying model text. That text is then the reply and is forwarded; a tool call consumes it instead.
When an audio-modality realtime model closes a turn with text and no audio, the framework falls back to TTS if one is configured and otherwise only logs an error while the caller hears silence. Issue one instructions- only reply asking for the same answer aloud, through the path tool replies already use. A retry that is again text-only ends there; a TTS fallback or a tool call on the turn means no retry.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 3 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| respeak_expected = False | ||
| if ( | ||
| unspoken_text_reply | ||
| and not tool_reply_expected | ||
| and not function_calls | ||
| and instructions != _RESPEAK_INSTRUCTIONS | ||
| ): | ||
| # nothing voiced the text reply; ask the model once for the same reply aloud | ||
| respeak_expected = True | ||
| speech_handle._num_steps += 1 | ||
| self._create_speech_task( | ||
| self._realtime_reply_task( | ||
| speech_handle=speech_handle, | ||
| model_settings=ModelSettings(tool_choice="none"), | ||
| instructions=_RESPEAK_INSTRUCTIONS, | ||
| ), | ||
| speech_handle=speech_handle, | ||
| name="AgentActivity.realtime_respeak", | ||
| ) | ||
| self._schedule_speech(speech_handle, SpeechHandle.SPEECH_PRIORITY_NORMAL, force=True) |
There was a problem hiding this comment.
🔴 Original answer lost after failed retry
When a text-only reply triggers a retry, respeak_expected omits its answer from history before the retry succeeds. If the retry fails or is interrupted, the answer disappears from local conversation history.
Learn more
The initial text-only reply is forwarded, but this branch excludes it from the local assistant history and speech items. The retry runs later as a separate task in _realtime_reply_task. That task can fail before returning a generation or stop after an interruption, leaving no replacement assistant message. Gemini also cannot remove the original from its server-side history, so the local and provider histories then disagree.
Example: Gemini returns "The menu is on the left" as text. The retry times out waiting for a generation. The session records neither that answer nor a spoken replacement, although the original reply was generated.
Recommended fix: Keep the original answer until the retry has produced a usable spoken response. If the retry fails, is interrupted before speaking, or returns no playable message, commit the original reply to local history and speech items; only discard it once a replacement is confirmed. Check how this interacts with Gemini's append-only server history.
Was this helpful? React with 👍 or 👎 to provide feedback.
| # model-turn text on a tool turn is the call's rendering, never a spoken reply | ||
| gen.unspoken_text = "" |
There was a problem hiding this comment.
🟡 Combined tool turns expose call rendering
When completion and a tool call share one server message, unspoken_text is cleared after the receive loop publishes it. Tool-call rendering then enters captions and assistant history as a reply.
Learn more
The receive loop handles server_content before tool_call from the same LiveServerMessage. A generation_complete or turn_complete flag closes the streams and converts held model text into reply text. _handle_tool_calls then clears the held value, but the text channel and modality future are already finalized.
Example: One server message carries model text call:getWeather{...}, generation_complete=True, and a getWeather tool call. The text is emitted and stored before this assignment runs, instead of remaining hidden as the tool call's rendering.
Recommended fix: Clear or classify held model text before processing completion whenever the same server message contains a tool call. Reorder the receive handling or pass tool-call presence into _handle_server_content so _close_output_streams cannot publish tool rendering.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if not current_gen.modalities.done(): | ||
| current_gen.modalities.set_result(["audio", "text"]) |
There was a problem hiding this comment.
🟡 Rejected audio suppresses text fallback
When an inline audio payload is invalid, modalities resolves to audio before frame validation discards it. Held text then skips text fallback, leaving the turn silent.
Learn more
The modality future is the definitive signal consumed by the voice layer. These lines mark the turn as audio before checking that part.inline_data.data is bytes and before constructing or sending an AudioFrame. The following ValueError handler keeps the session alive, but stream closure sees a completed modality future and cannot recover unspoken_text.
Example: Gemini sends held text Okay. plus an odd or non-byte inline payload that AudioFrame rejects. The error is logged and no frame reaches audio_ch, yet the message still reports audio and Okay. remains suppressed.
Recommended fix: Resolve the audio modality only after a valid frame is successfully created and queued. Track successful audio delivery rather than receipt of an inline-data part.
Was this helpful? React with 👍 or 👎 to provide feedback.
…follows The error fired as soon as a text-only message arrived in an audio session, before the respeak decision, so a turn the model then spoke still logged an error blaming chat context sync.
The emulation (rejecting tool calls emitted during the turn) is the supported behavior, and each rejection already logs when one happens. The framework sets tool_choice='none' routinely (final reply after max tool steps, realtime respeak), so the warning fired on normal turns.
…respeaking The text-only message and the respeak that voices it were both recorded as assistant messages, so the history held the same reply twice. Decide the respeak before recording the generation's messages and leave the unspoken text out when one follows; with no respeak it is still recorded.
|
Before/after audio of one call with Before (base gemini_text_only_before.mp4After (this PR, gemini_text_only_after.mp4 |
This branch has not been deployed
1 participant
Problem
Gemini Live in audio modality occasionally closes a turn with a text part and no audio. In one customer deployment (
gemini-live-2.5-flash-native-audioon Vertex AI, livekit-agents 1.6.10) a caller's "Okay, iska menu" was answered with a 68 ms generation of 5 text tokens and 0 audio tokens; the agent then sat silent until the caller hung up.Two things made that silence total:
modalitiesfuture as["audio", "text"]the moment a generation started, from the session config rather than from the content. The framework already handles a text-only realtime message (_process_one_message: TTS fallback when a TTS is configured, an error log otherwise), but it never saw one from Gemini.Change
fix(google)— resolve the message modality from what the turn carries. Audio on the first inline frame; text when the turn completes without audio while holding model text, in which case the text is forwarded as the reply and a warning is logged (Gemini answered an audio-modality turn with text only; nothing was spoken). A tool call on the turn consumes the held text, since it was the call's rendering. Text-modality sessions resolve eagerly as before.fix(voice)— when an audio-capable realtime model answers in text and no TTS voiced it, schedule one instructions-only reply asking for the same answer aloud, through the path tool replies already use, withtool_choice="none". A retry that is again text-only is not retried; an interruption or a tool call on the turn means no retry. With a TTS configured the existing fallback speaks the text and no retry happens.Tests
tests/test_plugin_google_realtime.py: a text-only audio turn resolves as["text"]and forwards its text; an audio turn stays unresolved until its first frame.tests/test_realtime_text_only_reply.py: without TTS exactly one respeakgenerate_replyis issued and the retry's audio is spoken; a text-only retry is not retried; with a TTS the text is spoken and no retry is issued.ruff,mypy -p livekit.plugins.google,mypy -p livekit.agents.voiceclean.Not in this PR
plugins/google/src/realtime/realtime_api.ts,Promise.resolve(modalities)in the generation start) and its agent activity the same fallback and error branch. Both changes need mirroring there.