Skip to content

Reconnect when the workload wakes from hibernation - #201

Merged
vitramir merged 1 commit into
mainfrom
wake-on-sigcont
Oct 31, 2026
Merged

vitramir merged 1 commit into
mainfrom
wake-on-sigcont

Conversation

@vitramir

Copy link
Copy Markdown
Contributor

The agynd half of workload hibernation's wake path, as specified in agynd-cli.md § 8 Wake (agynio/architecture#192). The turn-boundary report (§ 7) is a separate PR.

A hibernated workload resumes with every connection it held — Gateway, Notifications, LLM Proxy — dead on the far side, which cannot tell a frozen peer from a vanished one. The Notifications stream is the one that would wait on its dead peer until TCP gave up. On a wake agynd drops the subscription and reopens it, reads the inbox once with GetUnackedInboxItems (what arrived during the sleep is in it, and the stream that would have announced it died with the sleep), and touches a turn in flight now rather than at the next tick. Nothing is replayed to the agent CLI; the unary calls fail and reconnect on their own.

Two things say the workload has been woken:

  • The runner's SIGCONT, sent to the container's init once the guest's clock is set. A process with no handler never sees it, so main asks for it and hands the channel to Run.
  • The clock. The runner sets the wall clock after every restore while the monotonic clock slept through the pause, so the keepalive loop compares the two between ticks; a wall clock more than a tick ahead of monotonic time is a restore. In Go a Time carries both readings and Sub prefers the monotonic one, so the wall side is stripped with Round(0) before comparing — compared as they come, the two could never differ.

The keepalive loop handles the wake rather than Run, which is inside the turn for as long as a turn takes; a wake mid-turn must not wait for the turn to end. The handling is idempotent and coalesced: the runner signals twice a few seconds apart (a signal in the first half second after a resume can go missing), and the clock confirms what the signal announced. A repeat within two ticks of a handled wake finds it handled and does nothing, and one pending inbox read is as good as any number. The window is measured on the wall clock: the monotonic one does not move across a sleep, so two wakes a few seconds of running time apart but with a sleep between them are both handled.

A wake that lands while the subscriber is waiting out a failure cuts the wait short — the wait was for the failure, and a wake is not one. In holder mode the signal is logged and nothing else: a sandbox holds no subscription and runs no turn, and its gateway connection reopens itself on the next call.

Tests cover the signal, the clock jump, a second sleep as a second wake, the repeats of one wake folding into it, the mid-turn resubscribe and touch, and the backoff cut. go build, go vet, gofmt and go test ./... (also under -race) pass.

A hibernated workload resumes with every connection it held dead on the
far side, which cannot tell a frozen peer from a vanished one. The
Notifications stream is the one that would wait on its dead peer until
TCP gave up, so on a wake the subscription is dropped and reopened, the
inbox is read once -- what arrived during the sleep is in it, and the
stream that would have announced it died with the sleep -- and a turn
in flight is touched now rather than at the next tick. Nothing is
replayed to the agent CLI.

Two things say the workload has been woken. The runner sends SIGCONT to
the container's init once the guest's clock is set, which a process
without a handler never sees, so agynd asks for it. And the clock says
so on its own: the runner sets the wall clock after every restore while
the monotonic clock slept through the pause, so the keepalive compares
the two between ticks, and a wall clock more than a tick ahead is a
restore. The signal makes the wake prompt; the clock makes it certain.

The keepalive loop handles the wake rather than Run, which is inside
the turn for as long as a turn takes. The runner signals twice, a few
seconds apart, and the clock confirms what the signal announced; a
repeat within two ticks of a handled wake finds it handled and does
nothing, and one pending inbox read is as good as any number. The two
ticks are measured on the wall clock, because the monotonic one does
not move across a sleep and two wakes with a sleep between them must
both count.

A wake that lands while the subscriber is waiting out a failure cuts
the wait short: the wait was for the failure, and a wake is not one.
@vitramir
vitramir merged commit 1a3d7b4 into main Oct 31, 2026
1 of 2 checks passed
vitramir added a commit that referenced this pull request Oct 31, 2026
* Report the turn boundary to Runners

A turn used to end in silence. The keepalive stops when the agent CLI
goes quiet, and Runners learned of it only when the activity sweep
found the workload past its keepalive grace, which is how long any
decision that waits on an idle agent -- hibernation first among them --
had to wait as well.

After the turn's inbox items are acked, agynd now calls
ReportWorkloadIdle. The call flips the workload's agent_state to idle
at once, and is the moment the Orchestrator may hibernate the workload:
between one turn and the next, if the inbox is still empty when it
looks. The report follows the ack and never precedes it, so the ack is
the last word of a turn and a workload is never frozen between finishing
an item and acknowledging it. Every handler acks as its last act and
returns nil only once it has; the report sits behind that return, where
it is the same for codex, claude and agn.

A report that never arrives costs nothing but latency, because the
sweep reaches the same state on its own schedule, so a failure is
logged and never fatal. In holder mode there is no turn, so there is no
report; a sandbox's idleness is the Terminal Proxy's to observe.

The RPC is on the Gateway's RunnersGateway since agynio/api#199. The
repository commits no generated code; CI generates the bindings from
the BSR module, as before.

* Reconnect when the workload wakes from hibernation (#201)

A hibernated workload resumes with every connection it held dead on the
far side, which cannot tell a frozen peer from a vanished one. The
Notifications stream is the one that would wait on its dead peer until
TCP gave up, so on a wake the subscription is dropped and reopened, the
inbox is read once -- what arrived during the sleep is in it, and the
stream that would have announced it died with the sleep -- and a turn
in flight is touched now rather than at the next tick. Nothing is
replayed to the agent CLI.

Two things say the workload has been woken. The runner sends SIGCONT to
the container's init once the guest's clock is set, which a process
without a handler never sees, so agynd asks for it. And the clock says
so on its own: the runner sets the wall clock after every restore while
the monotonic clock slept through the pause, so the keepalive compares
the two between ticks, and a wall clock more than a tick ahead is a
restore. The signal makes the wake prompt; the clock makes it certain.

The keepalive loop handles the wake rather than Run, which is inside
the turn for as long as a turn takes. The runner signals twice, a few
seconds apart, and the clock confirms what the signal announced; a
repeat within two ticks of a handled wake finds it handled and does
nothing, and one pending inbox read is as good as any number. The two
ticks are measured on the wall clock, because the monotonic one does
not move across a sleep and two wakes with a sleep between them must
both count.

A wake that lands while the subscriber is waiting out a failure cuts
the wait short: the wait was for the failure, and a wake is not one.

* Run the holder test with the wake channel Run now takes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant