Added codex-temporal: a Temporal-backed durable executor. - #7
Open
moedash wants to merge 9 commits into
Open
Conversation
One Codex turn is one activity, and the rollout under ~/.codex/sessions stays the record. The workflow holds the thread id Codex mints, so a later turn resumes the same thread and a crash re-drives the turn in flight instead of starting over.
The heartbeat ran on a timer, so a codex that stopped producing events without dying looked healthy forever and the workflow waited behind it. Temporal recovers from a crash on its own; a hang is the shape that needs help. The activity now heartbeats on progress, kills the child once the gap passes CODEX_STALL_TIMEOUT_MS, and says so, which also frees the thread's lock for the next attempt. Activity cancellation kills the child too, so an interrupt no longer leaves codex running against the thread.
A worker killed outright leaves its codex holding the thread's writer lock, and every later attempt fails with "already has an active writer" until that process finishes. A retry now kills the holder first, but only when it has been reparented, which happens exactly when the worker that spawned it is gone. A codex driven by another live worker fails that test and is left alone. Worth noting a codex mid-stream dies on its own when the pipe closes, so this is the quiet case: the same hang the stall timeout covers, seen from the other side.
A turn is the durable unit, so a step was invisible from outside without reading the rollout. The heartbeat now carries the items codex has finished and the last one's type, which is what `temporal workflow describe` shows.
The integrity hashes in a package-lock.json read as misspelled words, and any npm lockfile will trip on that sooner or later. pnpm-lock.yaml was already skipped through the *-lock.yaml pattern; npm's name matches neither of the existing ones.
The two crash tests are the point of this package, and reproducing them by hand took a page of instructions and a broad pkill that took down other people's workers. Ports and state sit off the defaults now, and every process these scripts start is tracked by pid.
The gaps section still listed the aborted wording and the exec hang as open, and this branch now sits on the fixes for both.
moedash
force-pushed
the
moe/codex-temporal
branch
from
August 20, 2026 22:20
c988764 to
801bc4c
Compare
moedash
changed the base branch from
moe/exec-continue
to
moe/codex-integration
August 20, 2026 22:21
A turn retried a hundred times with no classification, so a codex that could never run burned a hundred model calls and the thread reported the result as an interrupt. Attempts default to five, the cap and the turn timeout come from env, and a failure no retry can fix stops on the first one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
codex-temporal/, a Temporal-backed durable executor for the Codex CLI. One Codex turn is one Temporal activity, and the rollout under~/.codex/sessionsstays the record.Based on
moe/codex-integration, which carries the four upstream slices this branch needs in order to run:--continue(#4), the two panic fixes (#2 and #3), theSIGKILLescalation (#5) and the unknown-outcome wording (#6). Review those on their own PRs. Based on--continuealone, this branch could not pass its own tool-crash test: resuming a thread with a dangling custom tool call panics without #3, and the process then sits on the thread's writer lock.Why?
Third harness in the same line of work after the OpenCode and Pi forks: the agent's own loop runs under a durable executor, while the session record stays in the app's own log. Codex is the cleanest of the three, because
codex execis already one process per turn, so there is no long-lived agent object to rebuild after a crash.The workflow holds the thread id Codex mints on the first turn, which is what makes a later turn resume the same thread. On a retry the activity reads the rollout to decide what to do: never recorded means run it, recorded and answered means the answer is already there, recorded and unfinished means carry the turn on.
How did you test it?
Against a live Codex built from this branch, through the
dev/scripts, which run both crash tests end to end and print PASS or FAIL.Two turns on one thread, the second resuming the first. A worker killed mid-tool-call: the turn finished on a fresh worker on attempt 2, the prompt was not asked twice, and the tool that had already completed did not run again, which is the assertion that matters.
Worth knowing about the tests themselves. Both refuse to conclude anything from a turn that already finished, because the first version of them waited for the workflow to report a thread id, which only arrives with the activity's result. The kill therefore always landed after the turn was over, and every assertion passed for the wrong reason. They now find the rollout by a token in the prompt text and check the activity attempt afterwards.
Also covered: a codex that hangs without dying is killed and re-driven rather than reported healthy, an aborted turn leaves no child behind, and a codex that outlived its worker is reaped before the retry.
The README carries the findings, including the two upstream bugs this work turned up and why there is no step level.