You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bug(kubernetes): the sandbox main process never gets SIGTERM when its pods are terminated #5104
When a sandbox's pods are terminated (eviction, node drain, pod deletion), the boundary freezes
the workload instead of passing SIGTERM to the main process and waiting up to the grace period,
as Kubernetes expects of a container's PID 1. A process that handles SIGTERM, for example to
save its work before a node drain, gets no window to do it.
Observed
OpenShell v0.1.5 (bdb956f2), Kubernetes driver, EKS (kubelet v1.35.6). The main process is a
uvicorn server that logs Shutting down on SIGTERM.
Action
Main process
Window before the workload is frozen
Delete the workload pod (as eviction or drain does)
No SIGTERM. The agent container logs Sandbox Protocol connection lost; workload frozen pending authenticated recovery
0 s
Delete the supervisor pod
Gets SIGTERM, then the relay closes and the workload is frozen
0.24 s
In a drain both pods are evicted together, so the main process gets nothing. Both pods then
wait out the 30 s grace period (#5025).
Expected
The boundary sends SIGTERM to the main process and waits for it to exit, up to the pod's
grace period, then SIGKILLs it.
The supervisor keeps the relay and egress working until the workload has exited, even if
its own pod is terminating at the same time.
Re-measured on 0.1.8: the main process still gets no SIGTERM in a node drain. #5279 fixed the 30 s hang, but not this.
Setup: EKS (kubelet v1.35.9), gateway, supervisor and sandbox runtime 0.1.8, Kubernetes driver with one supervisor pod per sandbox. The main process is a uvicorn server that logs Shutting down on SIGTERM. A child of the main process logs every 250 ms whether it has received SIGTERM and whether an HTTPS request through the sandbox's egress succeeds. Each case runs twice on a fresh sandbox. Times are from the kubectl delete pod --wait=false.
0.2 s after the SIGTERM (run 1: killed 1.1 s after it, sandbox boundary received SIGTERM before supervisor confirmation)
+2.0 / +3.0 s
works at the last sample before the kill
Delete both pods at once (drain)
No
+0.74 / +0.80 s
+2.8 s
not measured
Compared with 0.1.5: both pods now exit within about 3 s instead of waiting out the 30 s grace period (#5279). The main process's window hasn't changed: none on eviction or drain, under about a second when only the supervisor pod goes.
Why it matters. We run long-lived agent workloads in sandboxes: one task can take 5 to 30 minutes of model calls and tool runs. When the cluster retires a node (autoscaler consolidation, a node group upgrade, a spot interruption), its pods are drained. A workload that gets SIGTERM can stop at a safe point, save its session and workspace to object storage (about 1–2 s for us), and continue on another node. Today it is frozen and killed without notice, so the task is lost and has to restart from scratch. The fix doesn't need a preStop hook or a new API: SIGTERM forwarded to the main process, plus network access while it exits.
On #5342. It gives the workload a 5 s TERM grace when the supervisor pod gets SIGTERM, which covers part of this. Two questions:
In a drain, the workload pod receives SIGTERM too, and the boundary freezes the workload at once. Is forwarding SIGTERM from the workload pod's side, with the same grace, planned?
Activity
juancarlosm commentedon Nov 14, 2026
Re-measured on 0.1.8: the main process still gets no SIGTERM in a node drain. #5279 fixed the 30 s hang, but not this.
Setup: EKS (kubelet v1.35.9), gateway, supervisor and sandbox runtime 0.1.8, Kubernetes driver with one supervisor pod per sandbox. The main process is a uvicorn server that logs
Shutting downon SIGTERM. A child of the main process logs every 250 ms whether it has received SIGTERM and whether an HTTPS request through the sandbox's egress succeeds. Each case runs twice on a fresh sandbox. Times are from thekubectl delete pod --wait=false.Sandbox Protocol connection lost; workload frozen pending authenticated recovery)sandbox boundary received SIGTERM before supervisor confirmation)Compared with 0.1.5: both pods now exit within about 3 s instead of waiting out the 30 s grace period (#5279). The main process's window hasn't changed: none on eviction or drain, under about a second when only the supervisor pod goes.
Why it matters. We run long-lived agent workloads in sandboxes: one task can take 5 to 30 minutes of model calls and tool runs. When the cluster retires a node (autoscaler consolidation, a node group upgrade, a spot interruption), its pods are drained. A workload that gets SIGTERM can stop at a safe point, save its session and workspace to object storage (about 1–2 s for us), and continue on another node. Today it is frozen and killed without notice, so the task is lost and has to restart from scratch. The fix doesn't need a preStop hook or a new API: SIGTERM forwarded to the main process, plus network access while it exits.
On #5342. It gives the workload a 5 s TERM grace when the supervisor pod gets SIGTERM, which covers part of this. Two questions:
Happy to re-run this against a build with #5342, or any other branch.