[Bug]: Actors stay DELETING forever when runsc's _pause state file is truncated or the sandbox record is missing #2011

What happened?

Under load, actors end up in ACTOR_STATE_DELETING for good. Every retried DeleteActor, including any_state=true, fails the same way on the node, and the worker stays allocated until the worker pod is deleted by hand. We hit two node-side causes that the fix in #1666 (skip a container runsc no longer lists) does not cover.

1. runsc's state file for the pause container is truncated. runsc fails reading it with _pause.state: unexpected end of JSON input and exits 128, and the actor then stays DELETING. We have not pinned down what truncates it:

• The likeliest cause is that runsc writes that file with a plain truncate-then-write (SaveLocked in runsc/container/state_file.go), and ateom runs runsc under the caller's context (cmd/ateom-gvisor/runsc.go), so a cancelled request can SIGKILL runsc mid-write and leave the file empty.
• On delete, cleanupContainers (cmd/ateom-gvisor/main.go:898) runs runsc state _pause, which fails on the empty file. It then asks isContainerAlreadyGone, which uses runsc list -quiet. That lists state files by name without parsing them, so _pause is still listed, the container is not treated as gone, and the error is returned. runsc delete -force fails on the same file. Every retry of TerminateWorkload fails the same way.

2. The node's sandbox record is missing. Seen during a bulk delete right after a run in which many actors had calls in flight:

rpc error: code = Internal desc = failed to read sandbox record during terminate (actor: benchmark/sb-0a305a02-b6f6-431a-9bd9-c1b7f59e2a00, actorUID: 5a86c5b7-2e35-4d5d-a55d-c32cc77da1d2): TERMINAL_FILE_SYSTEM_ERROR: while reading sandbox record: open /var/lib/ateom-gvisor/actors/5a86c5b7-2e35-4d5d-a55d-c32cc77da1d2/sandbox-assets.json: no such file or directory

• atelet Terminate reads the record before anything else (cmd/atelet/main.go:1280) and fails if it is missing. The only thing that removes it is Terminate's own removeActorDirs, and Restore writes it only after ateom's RestoreWorkload succeeds (cmd/atelet/main.go:1260).
• So a Terminate that already ran on the node, followed by a failure of a later delete step or a lost lease, makes every retry fail. A restore that failed before the record was written does the same.
• The actor had in-flight calls when it was deleted. Just before, our other DeleteActor calls were refused with Aborted: another operation is in progress for this actor.

#1884 proposed a fix for case 2 and was closed in favor of sending the sandbox assets in the Terminate request. I could not find a follow-up issue or PR for that, so this issue also serves to track it.

Expected Behavior

DeleteActor on an actor whose node-side state is already partly torn down, or unreadable, should finish the teardown and free the worker:

• Terminate should not depend on a node-local record that an earlier attempt may have removed (the direction chosen on Make atelet.Terminate idempotent and write the sandbox record before Restore calls ateom #1884), or it should treat a missing record as nothing to terminate while still unmounting and removing the actor's directories.
• When runsc cannot read a container's state file, cleanup should kill the sandbox by PID and treat the container as gone. The actor's runsc-state directory is removed afterwards anyway.
• State-changing runsc commands should run on a context that a cancelled client request cannot kill mid-write (related: Traffic during suspend wedges actors in SUSPENDING and pins workers: -allow-connected-on-save missing on restore; post-checkpoint tail dies on caller ctx #1914).

Steps to Reproduce

We see this from a load test, not a minimal repro: 100 gVisor actors on one pool, each cycled resume / instruction / suspend for 5 minutes, then deleted in bulk with instructions still in flight. A few actors per run fail runsc restore with exit 128, and on some runs one or two end up in DELETING and stay there. The harness is ai-on-gke/agent-infra-benchmarking.

To recover by hand today, delete the worker pod. The actor moves to CRASHED, and the next DeleteActor succeeds.

Sandbox Runtime

gVisor (runsc)

Version

6ec93a4. The code paths above are the same on main at 12cf471 (2026-09-30).

Related

#1665 (umbrella for stuck transitional states), #372 (idempotent atelet API), #1884, #1953, #1518, #1914.