Repository navigation
Reconnect when the workload wakes from hibernation - #201
Merged
Merged
Conversation
A hibernated workload resumes with every connection it held dead on the far side, which cannot tell a frozen peer from a vanished one. The Notifications stream is the one that would wait on its dead peer until TCP gave up, so on a wake the subscription is dropped and reopened, the inbox is read once -- what arrived during the sleep is in it, and the stream that would have announced it died with the sleep -- and a turn in flight is touched now rather than at the next tick. Nothing is replayed to the agent CLI. Two things say the workload has been woken. The runner sends SIGCONT to the container's init once the guest's clock is set, which a process without a handler never sees, so agynd asks for it. And the clock says so on its own: the runner sets the wall clock after every restore while the monotonic clock slept through the pause, so the keepalive compares the two between ticks, and a wall clock more than a tick ahead is a restore. The signal makes the wake prompt; the clock makes it certain. The keepalive loop handles the wake rather than Run, which is inside the turn for as long as a turn takes. The runner signals twice, a few seconds apart, and the clock confirms what the signal announced; a repeat within two ticks of a handled wake finds it handled and does nothing, and one pending inbox read is as good as any number. The two ticks are measured on the wall clock, because the monotonic one does not move across a sleep and two wakes with a sleep between them must both count. A wake that lands while the subscriber is waiting out a failure cuts the wait short: the wait was for the failure, and a wake is not one.
This was referenced Oct 30, 2026
vitramir
added a commit
that referenced
this pull request
Oct 31, 2026
* Report the turn boundary to Runners A turn used to end in silence. The keepalive stops when the agent CLI goes quiet, and Runners learned of it only when the activity sweep found the workload past its keepalive grace, which is how long any decision that waits on an idle agent -- hibernation first among them -- had to wait as well. After the turn's inbox items are acked, agynd now calls ReportWorkloadIdle. The call flips the workload's agent_state to idle at once, and is the moment the Orchestrator may hibernate the workload: between one turn and the next, if the inbox is still empty when it looks. The report follows the ack and never precedes it, so the ack is the last word of a turn and a workload is never frozen between finishing an item and acknowledging it. Every handler acks as its last act and returns nil only once it has; the report sits behind that return, where it is the same for codex, claude and agn. A report that never arrives costs nothing but latency, because the sweep reaches the same state on its own schedule, so a failure is logged and never fatal. In holder mode there is no turn, so there is no report; a sandbox's idleness is the Terminal Proxy's to observe. The RPC is on the Gateway's RunnersGateway since agynio/api#199. The repository commits no generated code; CI generates the bindings from the BSR module, as before. * Reconnect when the workload wakes from hibernation (#201) A hibernated workload resumes with every connection it held dead on the far side, which cannot tell a frozen peer from a vanished one. The Notifications stream is the one that would wait on its dead peer until TCP gave up, so on a wake the subscription is dropped and reopened, the inbox is read once -- what arrived during the sleep is in it, and the stream that would have announced it died with the sleep -- and a turn in flight is touched now rather than at the next tick. Nothing is replayed to the agent CLI. Two things say the workload has been woken. The runner sends SIGCONT to the container's init once the guest's clock is set, which a process without a handler never sees, so agynd asks for it. And the clock says so on its own: the runner sets the wall clock after every restore while the monotonic clock slept through the pause, so the keepalive compares the two between ticks, and a wall clock more than a tick ahead is a restore. The signal makes the wake prompt; the clock makes it certain. The keepalive loop handles the wake rather than Run, which is inside the turn for as long as a turn takes. The runner signals twice, a few seconds apart, and the clock confirms what the signal announced; a repeat within two ticks of a handled wake finds it handled and does nothing, and one pending inbox read is as good as any number. The two ticks are measured on the wall clock, because the monotonic one does not move across a sleep and two wakes with a sleep between them must both count. A wake that lands while the subscriber is waiting out a failure cuts the wait short: the wait was for the failure, and a wake is not one. * Run the holder test with the wake channel Run now takes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The agynd half of workload hibernation's wake path, as specified in agynd-cli.md § 8 Wake (agynio/architecture#192). The turn-boundary report (§ 7) is a separate PR.
A hibernated workload resumes with every connection it held — Gateway, Notifications, LLM Proxy — dead on the far side, which cannot tell a frozen peer from a vanished one. The Notifications stream is the one that would wait on its dead peer until TCP gave up. On a wake agynd drops the subscription and reopens it, reads the inbox once with
GetUnackedInboxItems(what arrived during the sleep is in it, and the stream that would have announced it died with the sleep), and touches a turn in flight now rather than at the next tick. Nothing is replayed to the agent CLI; the unary calls fail and reconnect on their own.Two things say the workload has been woken:
SIGCONT, sent to the container's init once the guest's clock is set. A process with no handler never sees it, somainasks for it and hands the channel toRun.Timecarries both readings andSubprefers the monotonic one, so the wall side is stripped withRound(0)before comparing — compared as they come, the two could never differ.The keepalive loop handles the wake rather than
Run, which is inside the turn for as long as a turn takes; a wake mid-turn must not wait for the turn to end. The handling is idempotent and coalesced: the runner signals twice a few seconds apart (a signal in the first half second after a resume can go missing), and the clock confirms what the signal announced. A repeat within two ticks of a handled wake finds it handled and does nothing, and one pending inbox read is as good as any number. The window is measured on the wall clock: the monotonic one does not move across a sleep, so two wakes a few seconds of running time apart but with a sleep between them are both handled.A wake that lands while the subscriber is waiting out a failure cuts the wait short — the wait was for the failure, and a wake is not one. In holder mode the signal is logged and nothing else: a sandbox holds no subscription and runs no turn, and its gateway connection reopens itself on the next call.
Tests cover the signal, the clock jump, a second sleep as a second wake, the repeats of one wake folding into it, the mid-turn resubscribe and touch, and the backoff cut.
go build,go vet,gofmtandgo test ./...(also under-race) pass.