Repository navigation
gateway: blue-green restart handover — successor serves while blue drains - #6
Merged
Merged
Conversation
…lue drains Restart no longer blocks new work on the drain. request_restart spawns a successor 'gateway run' child (same unit cgroup) before the drain: green boots with platform-lock takeover authority, binds the control socket name when ready (new sessions/relays land there), and flips systemd's MainPID via sd_notify so blue's exit never triggers Restart=always. Blue finishes its in-flight turns through the existing stop path; the PID record and the host role are claimed by green's deferred task only after blue exits. Gate: agent.blue_green_restart (default off) + a Type=notify unit (NOTIFY_SOCKET), because the MainPID switch is the piece that makes Restart=always safe. No notify socket -> classic drain-exit restart, unchanged. Green-not-ready -> same fallback, logged. Single-owner services need no extra deferral: green's adapter cold-connects carry the same takeover authority as --replace, fencing blue's pollers out of each platform lock. Failure convergence: green crashing after MainPID still respawns the unit normally; green wedged pre-notify recovers via the existing exit-75 retry path.
…config gate, loop-safe probe Review findings (opus, systemd-probe-verified) addressed: - finding 1: the gate read config.agent which GatewayConfig lacks — the feature never enabled. Now loaded like restart_drain_timeout (_load_blue_green_restart -> runner._blue_green_restart) with an HERMES_BLUE_GREEN_RESTART env override. - finding 3: NotifyAccess=main drops a successor's own MAINPID notify; blue now sends MAINPID=<green> after readiness, green never self-notifies. - finding 4: green carries no platform-lock takeover authority (takeover SIGTERMs the holder = blue mid-drain). Green's adapters fail the lock while blue holds it and reconnect when blue's stop releases it. - finding 5: the readiness probe ran blocking socket I/O on blue's loop (1.5s stalls, watchdog risk) — now via asyncio.to_thread, with an early exit when the successor process dies. - finding 6/8: readiness now requires identify state==ready (published after adapters/cron/housekeeping), not a bound socket; MAINPID only switches after that, so green's watchdog is already feeding. - finding 7/13: fallback terminates the unpromoted successor; stop() terminates any unpromoted child; the spawn is skipped once a signal stop was requested. - finding 11: a lost deferred PID-record claim is fatal for green (self-SIGTERM) instead of serving unrecorded. - finding 15: handover env vars are scrubbed and the parse cached. Open (design note to follow on the PR): profile-reservation/session-store handoff, control-socket affinity for parked approval turns, fixed-port listener handover (api_server EADDRINUSE is non-retryable), systemctl-stop during overlap. Flag stays default-off until those land.
Owner
Author
Review disposition (opus review, probe-verified findings) — commit 5be2fc8Fixed in this push
Open — needs the single-owner handoff protocol settled before enabling (flag stays default-off)
Plan: short design note for the protocol (reservation + locks + ports + affinity), reviewed, then the mechanical remainder and the test matrix. Until then this PR stays draft and the feature stays off. |
IEatCodeDaily
marked this pull request as draft
November 15, 2026 18:09
Owner
Author
|
Owner decision (logged): the deferral is approved — this PR stays in draft, and the session-ownership protocol (profile-reservation/session-store custody during overlap, control-socket affinity, fixed-port listeners) moves to a dedicated design session; the owner picks the slot. Until then: no CI reruns, no green-chasing. The findings disposition remains the source of truth for open items. |
IEatCodeDaily
marked this pull request as ready for review
November 16, 2026 00:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
hermes gateway restartbecomes successor-first (blue-green) when enabled: the running gateway (blue) spawns the next gateway (green) before draining. Green boots with platform-lock takeover authority, binds the control socket name when ready — new sessions and relays land on green immediately — while blue keeps serving its established connections until their turns finish, then exits through the existing stop path.The piece that makes
Restart=alwayssafe is the MainPID switch: green sendssd_notify("MAINPID=<pid>")once it serves (units areType=notifywhen the watchdog is configured), so blue's exit is routine cgroup cleanup, not a MainPID crash. The PID record and the host role are claimed by a deferred task on green that waits for blue's exit — never racing blue while it drains.Safety
agent.blue_green_restart(default off) andNOTIFY_SOCKETpresent (Type=notify). Without either, restarts are byte-for-byte today's drain-exit behavior._platform_lock_takeover_on_start— the same authority--replaceuses — so its adapter cold-connects fence blue's pollers out of each platform lock. No new deferral machinery.Tests
tests/gateway/test_handover.py— env contract (round-trip, self/invalid-pid rejection), gate matrix,MAINPID=payload over a real datagram socket, readiness probe (pid match / mismatch timeout), predecessor-exit arming with a real child process, PID-reuse guard, and the claim-vs-live-owner matrix. Lane workflowgateway-blue-green-restart.ymlruns it focused; full-repo CI also fires on the PR.Activation (operators)
agent.systemd_watchdog_secondsconfigured (→Type=notifyunit) + re-generated unitagent.blue_green_restart: truehermes gateway restart— first restart is classic; every one after is successor-first.