Skip to content

gateway: blue-green restart handover — successor serves while blue drains - #6

Merged
IEatCodeDaily merged 2 commits into
mainfrom
feat/gateway-blue-green-restart
Nov 16, 2026
Merged

IEatCodeDaily merged 2 commits into
mainfrom
feat/gateway-blue-green-restart

Conversation

@IEatCodeDaily

Copy link
Copy Markdown
Owner

What

hermes gateway restart becomes successor-first (blue-green) when enabled: the running gateway (blue) spawns the next gateway (green) before draining. Green boots with platform-lock takeover authority, binds the control socket name when ready — new sessions and relays land on green immediately — while blue keeps serving its established connections until their turns finish, then exits through the existing stop path.

The piece that makes Restart=always safe is the MainPID switch: green sends sd_notify("MAINPID=<pid>") once it serves (units are Type=notify when the watchdog is configured), so blue's exit is routine cgroup cleanup, not a MainPID crash. The PID record and the host role are claimed by a deferred task on green that waits for blue's exit — never racing blue while it drains.

Safety

  • Gate: agent.blue_green_restart (default off) and NOTIFY_SOCKET present (Type=notify). Without either, restarts are byte-for-byte today's drain-exit behavior.
  • Green not ready in 20s → logged fallback to the classic drain-exit restart. Green crashes after taking MainPID → systemd respawns the unit normally. Green wedged pre-notify → the existing exit-75 retry path converges (same recovery as any wedged owner).
  • Single-owner services (messaging bots, platform locks): green boots with _platform_lock_takeover_on_start — the same authority --replace uses — so its adapter cold-connects fence blue's pollers out of each platform lock. No new deferral machinery.
  • The deferred PID-record claim refuses to steal the record from a still-live recorded gateway (supervisor-race guard) and tolerates a refused host-role claim (degraded, not dead).

Tests

tests/gateway/test_handover.py — env contract (round-trip, self/invalid-pid rejection), gate matrix, MAINPID= payload over a real datagram socket, readiness probe (pid match / mismatch timeout), predecessor-exit arming with a real child process, PID-reuse guard, and the claim-vs-live-owner matrix. Lane workflow gateway-blue-green-restart.yml runs it focused; full-repo CI also fires on the PR.

Activation (operators)

  1. agent.systemd_watchdog_seconds configured (→ Type=notify unit) + re-generated unit
  2. agent.blue_green_restart: true
  3. hermes gateway restart — first restart is classic; every one after is successor-first.

…lue drains

Restart no longer blocks new work on the drain. request_restart spawns a
successor 'gateway run' child (same unit cgroup) before the drain: green
boots with platform-lock takeover authority, binds the control socket name
when ready (new sessions/relays land there), and flips systemd's MainPID
via sd_notify so blue's exit never triggers Restart=always. Blue finishes
its in-flight turns through the existing stop path; the PID record and the
host role are claimed by green's deferred task only after blue exits.

Gate: agent.blue_green_restart (default off) + a Type=notify unit
(NOTIFY_SOCKET), because the MainPID switch is the piece that makes
Restart=always safe. No notify socket -> classic drain-exit restart,
unchanged. Green-not-ready -> same fallback, logged.

Single-owner services need no extra deferral: green's adapter cold-connects
carry the same takeover authority as --replace, fencing blue's pollers out
of each platform lock. Failure convergence: green crashing after MainPID
still respawns the unit normally; green wedged pre-notify recovers via the
existing exit-75 retry path.
…config gate, loop-safe probe

Review findings (opus, systemd-probe-verified) addressed:
- finding 1: the gate read config.agent which GatewayConfig lacks — the
  feature never enabled. Now loaded like restart_drain_timeout
  (_load_blue_green_restart -> runner._blue_green_restart) with an
  HERMES_BLUE_GREEN_RESTART env override.
- finding 3: NotifyAccess=main drops a successor's own MAINPID notify;
  blue now sends MAINPID=<green> after readiness, green never self-notifies.
- finding 4: green carries no platform-lock takeover authority (takeover
  SIGTERMs the holder = blue mid-drain). Green's adapters fail the lock
  while blue holds it and reconnect when blue's stop releases it.
- finding 5: the readiness probe ran blocking socket I/O on blue's loop
  (1.5s stalls, watchdog risk) — now via asyncio.to_thread, with an early
  exit when the successor process dies.
- finding 6/8: readiness now requires identify state==ready (published
  after adapters/cron/housekeeping), not a bound socket; MAINPID only
  switches after that, so green's watchdog is already feeding.
- finding 7/13: fallback terminates the unpromoted successor; stop()
  terminates any unpromoted child; the spawn is skipped once a signal
  stop was requested.
- finding 11: a lost deferred PID-record claim is fatal for green
  (self-SIGTERM) instead of serving unrecorded.
- finding 15: handover env vars are scrubbed and the parse cached.

Open (design note to follow on the PR): profile-reservation/session-store
handoff, control-socket affinity for parked approval turns, fixed-port
listener handover (api_server EADDRINUSE is non-retryable), systemctl-stop
during overlap. Flag stays default-off until those land.
@IEatCodeDaily

Copy link
Copy Markdown
Owner Author

Review disposition (opus review, probe-verified findings) — commit 5be2fc8

Fixed in this push

# Finding Fix
1 (blocker) gate read config.agent, which GatewayConfig lacks — feature was dead code _load_blue_green_restart() alongside the other agent timeouts → runner._blue_green_restart; env override HERMES_BLUE_GREEN_RESTART
3 (blocker) NotifyAccess=main drops green's own MAINPID= — probe showed blue's exit SIGKILLs green green never self-notifies; blue sends MAINPID=<green> after readiness
4 (blocker) green's takeover authority SIGTERMs blue mid-drain green carries no takeover authority; adapters fail the lock while blue holds it and reconnect when blue's stop releases it
5 (major) blocking identify probe stalled blue's loop (watchdog risk) probe via asyncio.to_thread + early exit when the successor process dies
6/8 (major) "ready" = socket bound, not serving; early MainPID switch starves green's watchdog readiness requires identify state == "ready" (published after adapters/cron/housekeeping); MAINPID switches only then
7 (major) fallback left green running fallback terminates the unpromoted successor (SIGTERM → bounded SIGKILL)
11 (major) lost deferred claim = unrecorded gateway serving on claim loss is fatal for green: self-SIGTERM, supervisor's next boot takes over
12 (major, partial) systemctl stop during overlap unpromoted children are terminated by stop(); the promoted case is covered by MainPID semantics — the blue-mid-drain SIGKILL window after a promoted green exits is still open, see below
13 (minor) signal stop during the handover window left green behind spawn skipped once a signal stop was requested; stop() terminates any unpromoted child
15 (minor) handover env leaked to green's children env scrubbed + parse cached; host-record refresh nit noted

Open — needs the single-owner handoff protocol settled before enabling (flag stays default-off)

Plan: short design note for the protocol (reservation + locks + ports + affinity), reviewed, then the mechanical remainder and the test matrix. Until then this PR stays draft and the feature stays off.

@IEatCodeDaily
IEatCodeDaily marked this pull request as draft November 15, 2026 18:09
@IEatCodeDaily

Copy link
Copy Markdown
Owner Author

Owner decision (logged): the deferral is approved — this PR stays in draft, and the session-ownership protocol (profile-reservation/session-store custody during overlap, control-socket affinity, fixed-port listeners) moves to a dedicated design session; the owner picks the slot. Until then: no CI reruns, no green-chasing. The findings disposition remains the source of truth for open items.

@IEatCodeDaily
IEatCodeDaily marked this pull request as ready for review November 16, 2026 00:55
@IEatCodeDaily
IEatCodeDaily merged commit 5afb332 into main Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant