[Bug]: Codex notification handlers block the app-server stdout reader, so a busy permit makes the shared app-server drop its T3 connection #28697

What happened

The shared Codex app-server went silent three times in ~80 minutes on Oct 31 (17:29, 18:30 and 18:51 UTC), and again on Nov 1 (13:25 UTC) on a newer nightly. Each time:

• it was in the middle of streaming item/agentMessage/delta;
• Codex logged Failed to write to stdout: Resource temporarily unavailable (os error 11);
• it marked its only client connection as disconnected, and dropped every later response and notification while the process stayed alive.

T3 kept sending requests and waited forever. The consequences (stuck workers, a blocked worktree lease) are tracked in #16636. This issue is about the trigger on T3's side.

Diagnosis

Likely cause: read-loop backpressure.

• packages/effect-codex-app-server/src/protocol.ts (~L406–430) handles each stdout line before reading the next chunk, and it runs onNotification handlers inline.
• In CodexAdapterV2.ts, the item/started, item/completed, thread/closed and thread/goal/* handlers run under turnTerminalizationPermit.withPermits(1) (~L4528, L4694, L4835, L5157). That is one Semaphore(1) for all threads on the shared app-server.
• So any permit holder that waits stops T3 from reading stdout. Codex fills its non-blocking stdout socket and gets EAGAIN.

Not yet identified: which holder waits. The goal-turn turn/interrupt (~L4511) is ruled out, since it is forked. The terminalization paths (~L5964) are unverified.

Ruled out: event-loop blocking (t3_server_event_loop_delay_max ≤15 ms at the drops), host load (CPU ≤14 %) and cgroup throttling (flat counters, PSI 0).

Fix idea: dispatch handlers off the reader, through a queue or a fork, and/or never hold the permit across an await.

Steps to reproduce

Not yet reduced to a minimal repro.

1. A Claude orchestrator repeatedly delegate_tasks to Codex (reviews), with long streamed replies, on one shared app-server.
2. Within tens of minutes Codex logs os error 11 on stdout, then dropping message for disconnected connection for every message.
3. Every later request goes unanswered, and running turns get no more events.

Deterministic idea: make a permit holder sleep while another thread streams deltas.

Version

0.1.6-nightly.20261030.3940 (three times on Oct 31) and 0.1.7-nightly.20261031.4004 (Nov 1), both with codex-cli 0.166.0, so it is not specific to one nightly. Triage CLI: 0.1.6.

Environment

• Arch Linux x64, kernel 7.2.11; t3 serve as a systemd user service; web UI client on the LAN
• codex-cli 0.166.0 as the shared app-server; Claude Agent (claude-opus-5-5) as orchestrator

Evidence

Codex `logs_2.sqlite`, first drop:

2026-10-31 17:29:04 ERROR codex_app_server_transport::transport::stdio  Failed to write to stdout: Resource temporarily unavailable (os error 11)
2026-10-31 17:29:04 WARN  codex_app_server::transport  dropping message for disconnected connection: ConnectionId(0)

On Nov 1 (4004) it was the same: EAGAIN at 13:25:29, last message from Codex at 13:25:28.931, and `thread/start` ids 27 and 32 never answered. Timelines: https://github.com/pingdotgg/t3code/issues/16636#issuecomment-6547877689

Related issues

#16636 (consequences), #23854 (bounds control calls, not this trigger), #21321 (worker slots only), #19066, openai/codex#55242

Fix applied or workaround

Stop the wedged app-server: systemctl --user stop t3code-codex-<id>.scope. T3 then fails or retries the pending turns, and starts a fresh app-server on the next turn. No backend restart was needed. Delegating to Claude instead of Codex avoids it.

Filed by

Claude Opus 5.5 · Claude Code · orchestrated by T3 Code (t3 triage). AI-generated; reviewed by @esukram before submission