Skip to content

feat(voice): talk with Scient Agent from a conversation's first message - #916

Merged
yaacovcorcos merged 1 commit into
mainfrom
claude/voice-call-starts-agent-20261114
Nov 14, 2026
Merged

yaacovcorcos merged 1 commit into
mainfrom
claude/voice-call-starts-agent-20261114

Conversation

@yaacovcorcos

Copy link
Copy Markdown
Contributor

Platforms: the same on macOS, Windows and Linux. The call agent is started like Scient Agent's sign-in process, through resolveSpawnCommand; the audio is WebRTC in the window, as before. Tested live on macOS only.

What

You can now talk with Scient Agent from a conversation's first message. A call used to run inside the conversation's Scient Agent session, so it needed a first message ("Send a first message, then talk") and an agent still running, which ruled out calls after a restart or 30 idle minutes.

With Scient Agent 0.1.10 or newer (scient-agent#55):

  • The call runs in an agent process of its own (provider/scient/ScientAgentLiveVoiceHost.ts, a new liveVoiceHost seam on the Scient Agent instance), apart from any conversation's session.
  • Each spoken request is sent to the conversation as the user's message. On a draft, the window sends the first one through the composer's draft delivery, so it creates the conversation exactly like a typed first message. After that the server sends them, as before.
  • The server follows the conversation and reports back (scient/liveVoice/LiveVoiceTurnReporter.ts). It finds the run that the request's message started, ignoring a turn that was already running, and sends that run's earlier assistant messages as progress and its last one as the answer to speak, via live_report. A failed or stopped run says so instead.
  • The call is kept per conversation outside the composer (useScientLiveVoiceCall.ts), so the switch from draft to conversation doesn't hang up. Leaving the conversation still hangs up, after a 2-second grace.

An agent older than 0.1.10 keeps the call in the conversation's session, as before, and the button still asks for a first message there.

T3 files change only inside the existing SCIENT-FORK voice blocks of ChatComposer.tsx and ws.ts.

Tests

  • LiveVoiceTurnReporter.test.ts: progress then the answer; a turn already running when the request was queued is not spoken; a failed run says so.
  • liveVoiceAvailability.test.ts: an agent at 0.1.10 can call before the first message, 0.1.9 asks for one, and a missing ChatGPT sign-in is asked for first.
  • Root pnpm run typecheck, lint, fmt and knip pass, as do the seam check and provenance. The server liveVoice, OmpLiveVoice and RpcAuthorization suites pass, and so do the web liveVoice and voice unit suites.
  • Live, macOS, test app with a 0.1.10 agent built from chore(upstream): sync T3 through ac4780f4 #55, with a spoken clip played in as the microphone:
    1. On an empty draft, "Talk with Scient Agent" was enabled; the call connected.
    2. "Please ask the agent to list the files in the current folder" created the conversation as its first message. The agent ran it, and the call said "The current folder is empty." The call stayed up through the draft-to-conversation switch.
    3. A second request in the same call, "create a file named hello.txt containing hello", was sent by the server. The agent created the file and the call said "Done".
  • Not tested live: a call on a conversation whose agent stopped. It takes the same path as step 3: the request is an ordinary message, which starts the agent.

🤖 Generated with Claude Code

A voice call ran inside the conversation's Scient Agent session, so it
needed a first message and an agent still running: none on a new
conversation, none after a restart or 30 idle minutes.

With Scient Agent 0.1.10 or newer, the call runs in an agent process of
its own. Each spoken request is sent to the conversation as the user's
message: on a draft the window sends the first one, which creates the
conversation like a typed message; after that the server sends them.
The server follows the conversation and reports the request's turn back
with `live_report`, progress as commentary and the answer to speak. The
call is kept per conversation outside the composer, so the switch from
draft to conversation doesn't hang up; leaving it still does. An older
agent keeps the call in the conversation's session, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:XL labels Nov 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.5 KiB 5.5 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 4.2 KiB 4.2 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.3 KiB 1.3 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 22.3 KiB 22.3 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.5 KiB 5.5 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 4.2 KiB 4.2 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.3 KiB 1.3 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 22.7 KiB 22.7 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

All 64 scenario measurements match main.

All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded 12.0 KiB 12.0 KiB 0 B (0.0%) 12.7 KiB ✅
Codex short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn running, cold open HTTP decoded 281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅
Codex long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Codex long turn running, history page HTTP decoded 92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅
Codex long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Codex long turn finished, cold open HTTP decoded 381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, socket cold open WebSocket decoded 381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, work page HTTP decoded 186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅
Codex long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Codex idle thread resume WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell snapshot HTTP decoded 45.1 KiB 45.1 KiB 0 B (0.0%) 53.7 KiB ✅
Codex fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Codex fleet shell subscribe WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Codex fleet shell resume check rows 0 0 0 0 ✅
Codex stale shell reload HTTP decoded 45.1 KiB 45.1 KiB 0 B (0.0%) 53.7 KiB ✅
Codex stale shell reload rows 27 27 0 (0.0%) 27 ✅
Codex stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Codex stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex stale todo list cold open HTTP decoded 72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅
Codex stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Codex stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅
Claude short turn resume WebSocket decoded 12.2 KiB 12.2 KiB 0 B (0.0%) 12.7 KiB ✅
Claude short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn running, cold open HTTP decoded 283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅
Claude long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Claude long turn running, history page HTTP decoded 93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅
Claude long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Claude long turn finished, cold open HTTP decoded 384.1 KiB 384.1 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, socket cold open WebSocket decoded 384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, work page HTTP decoded 188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅
Claude long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Claude idle thread resume WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell snapshot HTTP decoded 45.6 KiB 45.6 KiB 0 B (0.0%) 53.7 KiB ✅
Claude fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Claude fleet shell subscribe WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Claude fleet shell resume check rows 0 0 0 0 ✅
Claude stale shell reload HTTP decoded 45.6 KiB 45.6 KiB 0 B (0.0%) 53.7 KiB ✅
Claude stale shell reload rows 27 27 0 (0.0%) 27 ✅
Claude stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Claude stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude stale todo list cold open HTTP decoded 73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅
Claude stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Claude stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅

Baseline: 51d04b9 · PR result: 406c441 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Bounded compact HTTP snapshot with live cursor.

  • Codex decoded thread snapshot: 88.2 KiB
  • Claude decoded thread snapshot: 88.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@yaacovcorcos
yaacovcorcos added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit 1c0e361 Nov 14, 2026
59 checks passed
@yaacovcorcos
yaacovcorcos deleted the claude/voice-call-starts-agent-20261114 branch November 14, 2026 13:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant