Skip to content

bug(kubernetes): the sandbox main process never gets SIGTERM when its pods are terminated #5104

Open

Description

@juancarlosm

Problem

When a sandbox's pods are terminated (eviction, node drain, pod deletion), the boundary freezes
the workload instead of passing SIGTERM to the main process and waiting up to the grace period,
as Kubernetes expects of a container's PID 1. A process that handles SIGTERM, for example to
save its work before a node drain, gets no window to do it.

Observed

OpenShell v0.1.5 (bdb956f2), Kubernetes driver, EKS (kubelet v1.35.6). The main process is a
uvicorn server that logs Shutting down on SIGTERM.

Action Main process Window before the workload is frozen
Delete the workload pod (as eviction or drain does) No SIGTERM. The agent container logs Sandbox Protocol connection lost; workload frozen pending authenticated recovery 0 s
Delete the supervisor pod Gets SIGTERM, then the relay closes and the workload is frozen 0.24 s

In a drain both pods are evicted together, so the main process gets nothing. Both pods then
wait out the 30 s grace period (#5025).

Expected

  1. The boundary sends SIGTERM to the main process and waits for it to exit, up to the pod's
    grace period, then SIGKILLs it.
  2. The supervisor keeps the relay and egress working until the workload has exited, even if
    its own pod is terminating at the same time.
  3. Once the main process exits, both pods exit promptly, which also covers bug(sandbox): on Kubernetes, deleting a sandbox waits out the pods' termination grace period #5025.

Related

Activity

  1. juancarlosm commented on Nov 14, 2026

    @juancarlosm
    Author

    Re-measured on 0.1.8: the main process still gets no SIGTERM in a node drain. #5279 fixed the 30 s hang, but not this.

    Setup: EKS (kubelet v1.35.9), gateway, supervisor and sandbox runtime 0.1.8, Kubernetes driver with one supervisor pod per sandbox. The main process is a uvicorn server that logs Shutting down on SIGTERM. A child of the main process logs every 250 ms whether it has received SIGTERM and whether an HTTPS request through the sandbox's egress succeeds. Each case runs twice on a fresh sandbox. Times are from the kubectl delete pod --wait=false.

    Action SIGTERM to the main process Workload frozen Container killed (137) Egress
    Delete the workload pod (eviction) No +0.79 / +0.81 s (Sandbox Protocol connection lost; workload frozen pending authenticated recovery) +2.8 s works until frozen
    Delete the supervisor pod Yes, +0.79 / +0.89 s 0.2 s after the SIGTERM (run 1: killed 1.1 s after it, sandbox boundary received SIGTERM before supervisor confirmation) +2.0 / +3.0 s works at the last sample before the kill
    Delete both pods at once (drain) No +0.74 / +0.80 s +2.8 s not measured

    Compared with 0.1.5: both pods now exit within about 3 s instead of waiting out the 30 s grace period (#5279). The main process's window hasn't changed: none on eviction or drain, under about a second when only the supervisor pod goes.

    Why it matters. We run long-lived agent workloads in sandboxes: one task can take 5 to 30 minutes of model calls and tool runs. When the cluster retires a node (autoscaler consolidation, a node group upgrade, a spot interruption), its pods are drained. A workload that gets SIGTERM can stop at a safe point, save its session and workspace to object storage (about 1–2 s for us), and continue on another node. Today it is frozen and killed without notice, so the task is lost and has to restart from scratch. The fix doesn't need a preStop hook or a new API: SIGTERM forwarded to the main process, plus network access while it exits.

    On #5342. It gives the workload a 5 s TERM grace when the supervisor pod gets SIGTERM, which covers part of this. Two questions:

    1. In a drain, the workload pod receives SIGTERM too, and the boundary freezes the workload at once. Is forwarding SIGTERM from the workload pod's side, with the same grace, planned?
    2. fix(supervisor): bound dedicated SIGTERM shutdown and stop recovering a lost boundary #5342 doesn't recover a boundary connection lost during the grace, so a workload's cleanup can lose its network halfway through a save. Could the grace keep egress up until the workload exits, or at least be configurable?

    Happy to re-run this against a build with #5342, or any other branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions