Skip to content

fix(sandbox): give the health probe a deadline so startup cannot hang - #294

Merged
quickbeard merged 1 commit into
mainfrom
fix/sandbox-health-probe-hang
Aug 22, 2026
Merged

quickbeard merged 1 commit into
mainfrom
fix/sandbox-health-probe-hang

Conversation

@quickbeard

Copy link
Copy Markdown
Owner

Problem

You were right that #293 didn't fix this — the disk clone was a real inefficiency, but it was not the cause of the slow starts. This is.

A sandbox intermittently sits on starting for ~5 minutes and then fails with sandbox server did not become healthy within 60000ms, while the guest inside it is perfectly healthy. Caught it live on the failing run:

sandbox starting   15:09:47
# 15:14:10 — my probe straight to the guest, with the run's own credentials:
#   GET /global/health  ->  200 in 2ms
sandbox failed     15:14:49   "did not become healthy within 60000ms"

Instrumenting the poll loop showed exactly where it dies — a probe that starts and never settles:

[diag] health attempt #12 start +1352ms
[diag] health #12 -> false after 4ms
[diag] health attempt #13 start +1427ms      <- no result line, ever

pollSandboxHealth awaits each probe in turn, and checkSandboxHealth's fetch had no deadline. The forwarder accepts the TCP connection and dials through to a guest that is still binding its port; when that connection is left hanging rather than refused, the probe never settles, every later probe queues behind it, and nothing is left to notice that the sandbox came up a second later. Reproduced roughly one run in four.

Fix

Each probe gets a 2s deadline, and the poll's abort signal is threaded through so ending the race also cancels the in-flight request. A healthy guest answers in single-digit milliseconds, so a live probe never notices the deadline. All three providers (vz, bwrap, wsl) share the helper and the defect, so all three are fixed.

Verification

Four consecutive real bun dev:desktop runs, versus ~1-in-4 hanging before:

run starting → ready
1 3.98s
2 3.72s
3 1.93s
4 1.92s

The ~4s runs are one probe hitting the deadline and the next succeeding — the recovery working as designed.

Two new tests in boot.test.ts pin it: a probe against a fetch that never answers must still settle, and the caller's signal must cancel an in-flight probe. Both were confirmed to fail against the unfixed code (they hang and time out) — unlike the sparseness test I dropped from #293, these observe the property under Bun.

bun test in packages/codev-sandbox: 168 pass, 2 skip. Typecheck clean.

Changelog

#293's entry claimed the clone fixed slow first starts. It didn't, so I trimmed that entry back to what it actually did (instant creation, disk space) and this fix gets its own entry. Fork-only package, so no DIVERGENCES row.

Still unexplained

The 60s start timeout logged its failure ~300s in, not ~60s. The hang made that harmless to diagnose around, and it is moot now that probes always settle, but a setTimeout firing 5× late is its own anomaly. I could not account for it and did not chase it further.

🤖 Generated with Claude Code

pollSandboxHealth awaits each probe in turn, so one request that opened
and never answered stalled every later attempt: the sandbox stayed
'starting' until the 60s start timeout, minutes later, despite the guest
being healthy seconds in. Reproduced roughly one run in four.

Give each probe a 2s deadline and thread the poll's abort signal through
so cancelling the race also cancels the in-flight request. All three
providers share the helper and the defect.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@quickbeard
quickbeard merged commit 7867e1e into main Aug 22, 2026
0 of 5 checks passed
@quickbeard
quickbeard deleted the fix/sandbox-health-probe-hang branch August 22, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant