fix(openai): replay function calls and their outputs on a realtime reconnection

[rosetta-livekit-bot]

Ports livekit/agents#7576.

The OpenAI realtime session now replays function calls and outputs after reconnecting, preserves completed function-call arguments in its remote chat-context mirror, and resends chat-context events that the lost connection never confirmed.

No source behavior or required infrastructure was omitted.

Source diff coverage

Source diff coverage

• livekit-agents/livekit/agents/llm/_realtime/openai.py: adapted to plugins/openai/src/realtime/realtime_model.ts. Ported function-call/output replay, completed argument mirroring, and unconfirmed chat-context event resend using the target's TypeScript Future and websocket event tracking. The target-specific reconnect cleanup was adapted to retain pending item futures so resent confirmations can settle the original waiter.
• tests/test_realtime/test_openai_realtime_model.py: adapted to plugins/openai/src/realtime/realtime_model.test.ts. Ported both source tests and their mock realtime server using Vitest and ws, including target lifecycle cleanup.
• Not applicable: none. Both source files have direct target counterparts.

Testing

• pnpm build (40/40 workspace tasks passed)
• pnpm --filter @livekit/agents-plugin-openai build
• pnpm exec vitest run plugins/openai/src/realtime/realtime_model.test.ts (55 passed)
• pnpm exec eslint -f unix plugins/openai/src/realtime/realtime_model.test.ts
• pnpm exec vitest run plugins/openai (210 passed, 2 skipped; existing STT suite cannot load the LFS-placeholder Silero ONNX file, and the unlocked OpenAI 6.49 dependency makes an existing TTS mock return 404)
• pnpm lint (blocked by existing repository lint/format errors under the available Node 20/pnpm 9 toolchain; the repository declares pnpm 11, which requires Node >=22.13)
• cue-cli validation attempted after a successful build, but the configured LiveKit credentials were rejected with HTTP 401 before the worker could register

Changeset

• Patch @livekit/agents-plugin-openai

---

Ported from livekit/agents#7576

Original PR description

The OpenAI realtime session replays the conversation into the new websocket with exclude_function_call=True. That flag drops function calls and their outputs, so the new conversation is missing every tool exchange.

Two failures follow:

• A tool that is still running when the connection is replaced sends its output for a call id the new conversation does not have. The server answers invalid_tool_call_id ("Tool call ID '...' not found in conversation."), update_chat_ctx times out after 5 s, and the model never gets the result. This is the error in Tools throw error when working with OpenAI Realtime API agents#3344.
• Finished tool results disappear from the model's context after every scheduled replacement (max_session_duration, 20 min by default) and after every retried connection drop.

This PR makes three changes to the reconnection:

• It removes exclude_function_call=True from the replay. update_chat_ctx already sends function calls during a session, and the Azure Voice Live plugin already replays them on reconnection (test_reconnect_replays_tool_calls_and_system_messages).
• The remote chat context keeps each function call's completed arguments. conversation.item.added carries a model call with arguments: "", and only response.output_item.done carries the generated arguments. _handle_response_output_item_done now copies them into the mirrored FunctionCall, so the replayed call matches the one the tool ran with.
• The replay holds only the items the server confirmed. An item create or delete sent on the lost connection before its conversation.item.added or conversation.item.deleted arrived was missing from the new conversation, and update_chat_ctx timed out waiting for it. The session now keeps the in-flight chat context events it sent, and the reconnection sends the unconfirmed ones again after the replay. The new connection's confirmation settles the waiter. The Azure Voice Live plugin does the same (test_unconfirmed_items_are_created_again_after_a_reconnection).

Against the real OpenAI Realtime API with the first two changes applied to 1.8.2 (gpt-realtime, max_session_duration=10, a tool that took 15 s):

	Before	After
Server reply to the tool output	Tool call ID 'call_VMA0sMY2zSuVjwMt' not found in conversation.	conversation.item.added for the output of call_52xmy0WGEOqYIP3Q, after two reconnections while the tool ran
update_chat_ctx	update_chat_ctx timed out.	completed
Model reply after the tool	none	"You're in the goals space now. ..."
Each later reconnection	the call and its output are gone	the call and its output are replayed

A later run on the same setup hit the third case. The output was sent 0.5 s before the reconnection and was not confirmed, the replay carried only the call, and update_chat_ctx timed out.

Tests:

• test_reconnect_replays_function_calls_with_their_outputs fails before the first two changes (only the user message is replayed, and the call has empty arguments without the second change) and passes after them.
• test_reconnect_creates_an_item_the_lost_connection_never_confirmed closes the connection when the output arrives, before confirming it. It fails before the third change with update_chat_ctx timed out. and passes after it.

[changeset-bot]

🦋 Changeset detected

Latest commit: 68a0ba1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages

Name	Type
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-krisp	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-xai	Patch
@livekit/agents	Patch
@livekit/agents-plugin-anthropic	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-azure	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-meta	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-protoface	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR