Conversation
|
Pushed one more commit: It fixes a hole in this PR's own timeout. The timeout terminated the The consequence is worse than a slow probe. The fix escalates to Not covered by the suite: this path needs a live daemon, as does the timeout it fixes. Verified by hand against the exec that reproduced it. Static suite stays green (239 tests, 22 suites). |
|
Thank you for catching both of these, and for following up on the SIGKILL escalation. Can you add tests for early success during |
Thanks — all four are fair. start_period tests. Will add both: an early success inside the window marking the service healthy immediately, and a failure inside the window not consuming a retry. Hung container exec regression. Worth being precise about why this one matters. The existing test drives streamCommand with host sleep, and host sleep dies on SIGTERM — container exec does not, which is the whole reason the SIGKILL escalation is in this PR. So that test structurally cannot fail if the escalation is removed; it isn't weaker coverage of the same thing, it's coverage of a different process. I'll add the dynamic one alongside it rather than replacing it, since the static test still pins streamCommand's own timeout behaviour and runs without a daemon. One question before I write it: does CI have an Apple Container daemon available? The dynamic suite carries .containerDependent and skips when there's none, so if CI can't run it the regression would only ever execute locally. Either way is workable — I'd just rather it be a test you can rely on than one that silently skips. Explicit timeout tracking. Agreed, and it's a real hole: a process that exits 0 because it was signalled currently reads as a healthy probe. I'll carry the timeout in the result instead of inferring health from the exit status alone. |
The probe timeout terminated the `container exec` child with SIGTERM and assumed it would die. It does not: `container exec` ignores SIGTERM while the process it proxies is stuck. Measured against pgadmin's own healthcheck — the exec survived `timeout 25` untouched and only ended on SIGKILL (exit 137). The consequence was worse than a slow probe. `Process.terminationHandler` never fires, so the continuation never resumes, so `waitUntilServiceIsHealthy` never returns and the whole `up` stalls indefinitely, silently, with no error and no further output. Two runs died this way before the cause was found; both looked like a hung image pull. Escalate two seconds after the SIGTERM. A probe that overruns its timeout is then one failed attempt, which is what Docker does and what the retry loop below already assumes. Untested by the suite: this path needs a live daemon, as does the timeout it fixes. Verified by hand against the exec that reproduced it.
…policy
A healthcheck probe that installs a signal handler and exits 0 when terminated
reported the same exit status as one that genuinely passed, so a probe killed for
overrunning `healthcheck.timeout` could be recorded as healthy. The status alone
cannot answer the question, so `streamCommand` now returns a `CommandOutcome`
carrying both the status and whether the timeout fired, and the probe asks
`succeeded` rather than comparing to zero. The flag is set before the signal is
sent, so the race between terminating and exiting cannot hide it.
Regression, with no daemon involved: `sh -c 'trap "exit 0" TERM; sleep 30'` under
a 1s timeout gives `status == 0`, `timedOut == true`, `succeeded == false`. The
first two assertions are what the old return type could express; only the third
distinguishes this from a passing probe.
The `start_period` behaviour was already correct and had nothing holding it in
place. Its policy is now `awaitHealthy`, with the probe, the sleep and the clock
injected, which makes both properties assertable in a millisecond instead of by
waiting out real windows:
* a probe that succeeds inside the window returns immediately, without
sleeping — a service ready in one second does not sit out a 30s grace window
* failures inside the window do not consume the retry budget: a 3s window at
1s intervals against an always-failing probe gives 3 probes inside it and
then the full 3 retries, 6 calls rather than 3
The window stays wall clock rather than a count of iterations, so a slow probe
consumes it exactly as it does in production; `now` is injected only so a test
can drive that clock.
The static suite drives `streamCommand` with host `sleep`, and host `sleep` dies on SIGTERM. `container exec` does not: while the process it proxies is stuck it survives the signal, so a probe timeout that only calls `terminate()` never returns and `up` stalls rather than recording one failed attempt. No test built on a host process can tell those apart, which is why this one starts a container. It execs a 300s sleep under a 2s timeout and asserts the call comes back at all, that the outcome is marked as timed out, and that it is therefore not a success. Runs in about five seconds against a live daemon, and skips through `.containerDependent` where there is none. Kept alongside the host-`sleep` tests rather than replacing them: those pin `streamCommand`'s own timeout behaviour and need no daemon, so they still run where this one cannot.
The teardown was a `defer` spawning a detached Task, which does not hold the async call — measured: two stopped containers left behind after two runs. It is now an awaited function called on both the success and the throwing path. It also removes the container explicitly. `compose down` stops it but leaves it listed, so `down` alone accumulates one stopped container per run.
94f7f69 to
8a917a5
Compare
|
Pushed. All four are addressed; the branch is rebased onto current
The window stays wall clock rather than becoming a count of iterations, so a slow probe consumes it exactly as it does in production; Explicit timeout tracking. The regression is the case you were pointing at, and it needs no daemon: Hung Mutation-checked: removing the SIGKILL escalation makes it hang rather than fail — the call never returns, which is exactly the defect it guards. The host- Static suite: 250 tests / 24 suites green, up from 245 / 23. One thing I noticed while writing the teardown, unrelated to this PR and not addressed here: |
fix(up): enforce
healthcheck.timeoutand treatstart_periodas a grace windowSummary
depends_on: condition: service_healthyworks today (#61, #75, #117), but two of the fivehealthcheckfields do not behave as Compose specifies:timeoutis decoded and never applied. A probe that blocks stallsupindefinitely.start_periodis implemented as an unconditional pre-sleep, not as Compose's grace window.Every service that declares one pays it in full even when it is ready immediately.
Both are in
waitUntilServiceIsHealthy.1.
timeoutis parsed but never appliedHealthcheck.timeoutis decoded (Codable Structs/Healthcheck.swift:79) and has parsing tests,but
waitUntilServiceIsHealthy(Commands/ComposeUp.swift:1247) never reads it:streamCommand(Commands/ComposeUp.swift:1481) resumes its continuation only fromprocess.terminationHandler, so the probe is unbounded.grep -rn timeout Sources/returns hitsonly in
Healthcheck.swiftplus unrelated comments about the start wait.Reproducer
docker compose uphangunhealthy; fails in ~5s withdependency failed to startcontainer-compose uponmainupis silent for ~30 minutesThis is not synthetic. Healthchecks that talk to a socket block for real:
bash's
/dev/tcphas no connect timeout, so against a filtered (not refused) port it blocks untilthe kernel gives up.
timeout: 5sis exactly what is supposed to bound it.2.
start_periodis a pre-sleep, not a grace windowCompose semantics are the other way round: probes do run during
start_period, their failuresdo not consume
retries, and the first success ends the window immediately. Today the firstprobe cannot run until the whole window has elapsed, so
start_period: 40scosts 40 seconds evenfor a service that is ready in 200 ms.
upstarts services sequentially (#128 tracks parallelising it), so the cost is additive across theproject. Measured on a real 23-service compose file (YAML merge keys resolved, so anchor-inherited
values are counted): 18 services declare a healthcheck — 9 at
start_period: 10s, 4 at 15s, 1 at20s, 3 at 30s, 1 at 40s. That is 300 seconds of mandatory sleep before those services are first
probed, regardless of how quickly they actually become ready.
What this PR changes
streamCommandgains an optionaltimeout:. When it elapses and the child is still running theprocess is terminated; the termination handler then fires with a non-zero status, so the caller
sees an ordinary failed check. This matches Docker, where a timed-out probe counts as one failure
rather than as an error.
waitUntilServiceIsHealthypasseshealthcheck.timeout(Compose default 30s) to every probe.start_periodbecomes a grace window: probes run atintervalinside it, failures there do notconsume the retry budget, and the first success returns.
timeout:defaults tonil, so the two existingstreamCommandcall sites (run, and theforeground service stream) keep their current unbounded behaviour. No other call site changes.
Tests
New
Tests/Container-Compose-StaticTests/HealthcheckTimeoutTests.swift:streamCommandterminates a child that overruns its timeout:sleep 30withtimeout: 1returns non-zero in ~1.1 s. On
mainthe same call takes 30 s and returns 0.timeoutresolves to the Compose default of 30 s, not to "no timeout".These need no running
containerdaemon.Verification
Run on macOS 27.0 / Swift 6.4, branched from
main@6e6aaf0:swift buildswift test(static suites)Negative control: with
process.terminate()deleted from the new timeout branch, thestreamCommand terminates a child that overruns its timeouttest fails on both expectations andtakes 30.039 s instead of 1.090 s. The test does discriminate the fix from its absence.
Not covered: an end-to-end
container-compose upagainst a live daemon. Thestart_periodchangehas no unit test for the same reason, only the reasoning above and the existing dynamic suite.
Notes and limits
container execdoes not guarantee the process inside the guest isreaped. Docker has the same caveat with its own exec-based probes.
interval, so withinterval > start_periodit can overshoot thewindow by up to one interval. Docker behaves the same way.
start_periodas ahard floor to mask a startup race will now proceed earlier. That is the Compose-correct behaviour,
but it is worth a line in the release notes.
container recover. Container-Compose still treats the health wait as a one-shot gate during
up.That is a larger change and belongs with full support: depends_on #68.