Repository navigation
fix(sandbox): give the health probe a deadline so startup cannot hang - #294
Merged
Merged
Conversation
pollSandboxHealth awaits each probe in turn, so one request that opened and never answered stalled every later attempt: the sandbox stayed 'starting' until the 60s start timeout, minutes later, despite the guest being healthy seconds in. Reproduced roughly one run in four. Give each probe a 2s deadline and thread the poll's abort signal through so cancelling the race also cancels the in-flight request. All three providers share the helper and the defect. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
You were right that #293 didn't fix this — the disk clone was a real inefficiency, but it was not the cause of the slow starts. This is.
A sandbox intermittently sits on
startingfor ~5 minutes and then fails withsandbox server did not become healthy within 60000ms, while the guest inside it is perfectly healthy. Caught it live on the failing run:Instrumenting the poll loop showed exactly where it dies — a probe that starts and never settles:
pollSandboxHealthawaits each probe in turn, andcheckSandboxHealth's fetch had no deadline. The forwarder accepts the TCP connection and dials through to a guest that is still binding its port; when that connection is left hanging rather than refused, the probe never settles, every later probe queues behind it, and nothing is left to notice that the sandbox came up a second later. Reproduced roughly one run in four.Fix
Each probe gets a 2s deadline, and the poll's abort signal is threaded through so ending the race also cancels the in-flight request. A healthy guest answers in single-digit milliseconds, so a live probe never notices the deadline. All three providers (vz, bwrap, wsl) share the helper and the defect, so all three are fixed.
Verification
Four consecutive real
bun dev:desktopruns, versus ~1-in-4 hanging before:The ~4s runs are one probe hitting the deadline and the next succeeding — the recovery working as designed.
Two new tests in
boot.test.tspin it: a probe against a fetch that never answers must still settle, and the caller's signal must cancel an in-flight probe. Both were confirmed to fail against the unfixed code (they hang and time out) — unlike the sparseness test I dropped from #293, these observe the property under Bun.bun testinpackages/codev-sandbox: 168 pass, 2 skip. Typecheck clean.Changelog
#293's entry claimed the clone fixed slow first starts. It didn't, so I trimmed that entry back to what it actually did (instant creation, disk space) and this fix gets its own entry. Fork-only package, so no DIVERGENCES row.
Still unexplained
The 60s start timeout logged its failure ~300s in, not ~60s. The hang made that harmless to diagnose around, and it is moot now that probes always settle, but a
setTimeoutfiring 5× late is its own anomaly. I could not account for it and did not chase it further.🤖 Generated with Claude Code