Skip to content

fix(up): enforce healthcheck.timeout and treat start_period as a grace window - #151

Open
Mikimoto wants to merge 5 commits into
Mcrich23:mainfrom
Mikimoto:fix/healthcheck-timeout-start-period
Open

Mikimoto wants to merge 5 commits into
Mcrich23:mainfrom
Mikimoto:fix/healthcheck-timeout-start-period

Conversation

@Mikimoto

@Mikimoto Mikimoto commented Sep 1, 2026

Copy link
Copy Markdown

fix(up): enforce healthcheck.timeout and treat start_period as a grace window

Summary

depends_on: condition: service_healthy works today (#61, #75, #117), but two of the five
healthcheck fields do not behave as Compose specifies:

  1. timeout is decoded and never applied. A probe that blocks stalls up indefinitely.
  2. start_period is implemented as an unconditional pre-sleep, not as Compose's grace window.
    Every service that declares one pays it in full even when it is ready immediately.

Both are in waitUntilServiceIsHealthy.


1. timeout is parsed but never applied

Healthcheck.timeout is decoded (Codable Structs/Healthcheck.swift:79) and has parsing tests,
but waitUntilServiceIsHealthy (Commands/ComposeUp.swift:1247) never reads it:

let exitCode = try await streamCommand(
    "container",
    args: ["exec", containerName] + execArguments,
    onStdout: { _ in },
    onStderr: { _ in }
)

streamCommand (Commands/ComposeUp.swift:1481) resumes its continuation only from
process.terminationHandler, so the probe is unbounded. grep -rn timeout Sources/ returns hits
only in Healthcheck.swift plus unrelated comments about the start wait.

Reproducer

services:
  hang:
    image: alpine:3
    command: ["sleep", "infinity"]
    healthcheck:
      test: ["CMD", "sleep", "600"]
      interval: 1s
      timeout: 1s
      retries: 3
  app:
    image: alpine:3
    command: ["echo", "ok"]
    depends_on:
      hang:
        condition: service_healthy
behaviour
docker compose up each probe killed at 1s; 3 consecutive failures mark hang unhealthy; fails in ~5s with dependency failed to start
container-compose up on main first probe runs the full 600s, then repeats twice. up is silent for ~30 minutes

This is not synthetic. Healthchecks that talk to a socket block for real:

test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/5432 && exec 4<>/dev/tcp/127.0.0.1/5433"]

bash's /dev/tcp has no connect timeout, so against a filtered (not refused) port it blocks until
the kernel gives up. timeout: 5s is exactly what is supposed to bound it.


2. start_period is a pre-sleep, not a grace window

if startPeriod > 0 {
    try await Task.sleep(nanoseconds: UInt64(startPeriod * 1_000_000_000))
}

Compose semantics are the other way round: probes do run during start_period, their failures
do not consume retries, and the first success ends the window immediately. Today the first
probe cannot run until the whole window has elapsed, so start_period: 40s costs 40 seconds even
for a service that is ready in 200 ms.

up starts services sequentially (#128 tracks parallelising it), so the cost is additive across the
project. Measured on a real 23-service compose file (YAML merge keys resolved, so anchor-inherited
values are counted): 18 services declare a healthcheck — 9 at start_period: 10s, 4 at 15s, 1 at
20s, 3 at 30s, 1 at 40s. That is 300 seconds of mandatory sleep before those services are first
probed, regardless of how quickly they actually become ready.


What this PR changes

  • streamCommand gains an optional timeout:. When it elapses and the child is still running the
    process is terminated; the termination handler then fires with a non-zero status, so the caller
    sees an ordinary failed check. This matches Docker, where a timed-out probe counts as one failure
    rather than as an error.
  • waitUntilServiceIsHealthy passes healthcheck.timeout (Compose default 30s) to every probe.
  • start_period becomes a grace window: probes run at interval inside it, failures there do not
    consume the retry budget, and the first success returns.

timeout: defaults to nil, so the two existing streamCommand call sites (run, and the
foreground service stream) keep their current unbounded behaviour. No other call site changes.


Tests

New Tests/Container-Compose-StaticTests/HealthcheckTimeoutTests.swift:

  • streamCommand terminates a child that overruns its timeout: sleep 30 with timeout: 1
    returns non-zero in ~1.1 s. On main the same call takes 30 s and returns 0.
  • a child that finishes inside its timeout is left alone and its exit code preserved.
  • an omitted timeout resolves to the Compose default of 30 s, not to "no timeout".

These need no running container daemon.

Verification

Run on macOS 27.0 / Swift 6.4, branched from main @ 6e6aaf0:

check result
swift build 0 errors
swift test (static suites) 239 tests in 22 suites passed
new suite 3 tests passed in 1.14 s

Negative control: with process.terminate() deleted from the new timeout branch, the
streamCommand terminates a child that overruns its timeout test fails on both expectations and
takes 30.039 s instead of 1.090 s. The test does discriminate the fix from its absence.

Not covered: an end-to-end container-compose up against a live daemon. The start_period change
has no unit test for the same reason, only the reasoning above and the existing dynamic suite.


Notes and limits

  • Terminating the host-side container exec does not guarantee the process inside the guest is
    reaped. Docker has the same caveat with its own exec-based probes.
  • The grace-window loop probes at interval, so with interval > start_period it can overshoot the
    window by up to one interval. Docker behaves the same way.
  • Dropping the pre-sleep is a behaviour change: a compose file that leaned on start_period as a
    hard floor to mask a startup race will now proceed earlier. That is the Compose-correct behaviour,
    but it is worth a line in the release notes.
  • Not addressed here: Compose keeps probing after a service is healthy and lets an unhealthy
    container recover. Container-Compose still treats the health wait as a one-shot gate during up.
    That is a larger change and belongs with full support: depends_on #68.

@Mikimoto

Mikimoto commented Sep 9, 2026

Copy link
Copy Markdown
Author

Pushed one more commit: fix(up): escalate a timed-out healthcheck probe to SIGKILL.

It fixes a hole in this PR's own timeout. The timeout terminated the container exec child with SIGTERM and assumed it would die; it does not. container exec ignores SIGTERM while the process it proxies is stuck:

timeout 25 container exec <name> <probe>       # survives untouched
timeout -s KILL 20 container exec <name> <probe>   # exit 137

The consequence is worse than a slow probe. Process.terminationHandler never fires, so the continuation never resumes, so waitUntilServiceIsHealthy never returns — up stalls indefinitely with no error and no further output. I hit this twice while bringing a 23-service compose file up and both times mistook it for a hung image pull.

The fix escalates to kill(pid, SIGKILL) two seconds after the SIGTERM, so an overrunning probe becomes one failed attempt, which is what Docker does and what the retry loop below already assumes.

Not covered by the suite: this path needs a live daemon, as does the timeout it fixes. Verified by hand against the exec that reproduced it. Static suite stays green (239 tests, 22 suites).

@Mcrich23

Mcrich23 commented Sep 9, 2026

Copy link
Copy Markdown
Owner

Thank you for catching both of these, and for following up on the SIGKILL escalation. Can you add tests for early success during start_period and failures not consuming retries during that window? I'd also like a dynamic regression with a hung container exec, since the existing automated test uses host sleep. Please track timeout explicitly so a process that exits 0 after receiving the termination signal doesn't count as healthy.

@Mikimoto

Copy link
Copy Markdown
Author

Thank you for catching both of these, and for following up on the SIGKILL escalation. Can you add tests for early success during start_period and failures not consuming retries during that window? I'd also like a dynamic regression with a hung container exec, since the existing automated test uses host sleep. Please track timeout explicitly so a process that exits 0 after receiving the termination signal doesn't count as healthy.

Thanks — all four are fair.

start_period tests. Will add both: an early success inside the window marking the service healthy immediately, and a failure inside the window not consuming a retry.

Hung container exec regression. Worth being precise about why this one matters. The existing test drives streamCommand with host sleep, and host sleep dies on SIGTERM — container exec does not, which is the whole reason the SIGKILL escalation is in this PR. So that test structurally cannot fail if the escalation is removed; it isn't weaker coverage of the same thing, it's coverage of a different process. I'll add the dynamic one alongside it rather than replacing it, since the static test still pins streamCommand's own timeout behaviour and runs without a daemon.

One question before I write it: does CI have an Apple Container daemon available? The dynamic suite carries .containerDependent and skips when there's none, so if CI can't run it the regression would only ever execute locally. Either way is workable — I'd just rather it be a test you can rely on than one that silently skips.

Explicit timeout tracking. Agreed, and it's a real hole: a process that exits 0 because it was signalled currently reads as a healthy probe. I'll carry the timeout in the result instead of inferring health from the exit status alone.

tmp and others added 5 commits September 15, 2026 05:19
The probe timeout terminated the `container exec` child with SIGTERM and
assumed it would die. It does not: `container exec` ignores SIGTERM while
the process it proxies is stuck. Measured against pgadmin's own
healthcheck — the exec survived `timeout 25` untouched and only ended on
SIGKILL (exit 137).

The consequence was worse than a slow probe. `Process.terminationHandler`
never fires, so the continuation never resumes, so `waitUntilServiceIsHealthy`
never returns and the whole `up` stalls indefinitely, silently, with no
error and no further output. Two runs died this way before the cause was
found; both looked like a hung image pull.

Escalate two seconds after the SIGTERM. A probe that overruns its timeout
is then one failed attempt, which is what Docker does and what the retry
loop below already assumes.

Untested by the suite: this path needs a live daemon, as does the timeout
it fixes. Verified by hand against the exec that reproduced it.
…policy

A healthcheck probe that installs a signal handler and exits 0 when terminated
reported the same exit status as one that genuinely passed, so a probe killed for
overrunning `healthcheck.timeout` could be recorded as healthy. The status alone
cannot answer the question, so `streamCommand` now returns a `CommandOutcome`
carrying both the status and whether the timeout fired, and the probe asks
`succeeded` rather than comparing to zero. The flag is set before the signal is
sent, so the race between terminating and exiting cannot hide it.

Regression, with no daemon involved: `sh -c 'trap "exit 0" TERM; sleep 30'` under
a 1s timeout gives `status == 0`, `timedOut == true`, `succeeded == false`. The
first two assertions are what the old return type could express; only the third
distinguishes this from a passing probe.

The `start_period` behaviour was already correct and had nothing holding it in
place. Its policy is now `awaitHealthy`, with the probe, the sleep and the clock
injected, which makes both properties assertable in a millisecond instead of by
waiting out real windows:

  * a probe that succeeds inside the window returns immediately, without
    sleeping — a service ready in one second does not sit out a 30s grace window
  * failures inside the window do not consume the retry budget: a 3s window at
    1s intervals against an always-failing probe gives 3 probes inside it and
    then the full 3 retries, 6 calls rather than 3

The window stays wall clock rather than a count of iterations, so a slow probe
consumes it exactly as it does in production; `now` is injected only so a test
can drive that clock.
The static suite drives `streamCommand` with host `sleep`, and host `sleep` dies
on SIGTERM. `container exec` does not: while the process it proxies is stuck it
survives the signal, so a probe timeout that only calls `terminate()` never
returns and `up` stalls rather than recording one failed attempt. No test built
on a host process can tell those apart, which is why this one starts a container.

It execs a 300s sleep under a 2s timeout and asserts the call comes back at all,
that the outcome is marked as timed out, and that it is therefore not a success.
Runs in about five seconds against a live daemon, and skips through
`.containerDependent` where there is none.

Kept alongside the host-`sleep` tests rather than replacing them: those pin
`streamCommand`'s own timeout behaviour and need no daemon, so they still run
where this one cannot.
The teardown was a `defer` spawning a detached Task, which does not hold the
async call — measured: two stopped containers left behind after two runs. It is
now an awaited function called on both the success and the throwing path.

It also removes the container explicitly. `compose down` stops it but leaves it
listed, so `down` alone accumulates one stopped container per run.
@Mikimoto
Mikimoto force-pushed the fix/healthcheck-timeout-start-period branch from 94f7f69 to 8a917a5 Compare September 15, 2026 05:43
@Mikimoto

Copy link
Copy Markdown
Author

Pushed. All four are addressed; the branch is rebased onto current main (which now carries #144).

start_period tests. The policy is extracted as awaitHealthy, with the probe, the sleep and the clock injected, so both properties are assertable without waiting out real windows:

  • a probe that succeeds inside the window returns immediately and without sleeping — a service ready in one second no longer sits out a 30s grace window
  • failures inside the window don't consume the retry budget: a 3s window at 1s intervals against an always-failing probe gives 3 probes inside it and then the full 3 retries, 6 calls rather than 3

The window stays wall clock rather than becoming a count of iterations, so a slow probe consumes it exactly as it does in production; now is injected only so a test can drive that clock. Four tests, 1ms, no daemon. Verified they discriminate: disabling the grace window reddens one, and dropping the early return reddens three.

Explicit timeout tracking. streamCommand now returns a CommandOutcome carrying the exit status and whether the timeout fired, set before the signal is sent so the terminate/exit race can't hide it. The probe asks succeeded instead of comparing to zero.

The regression is the case you were pointing at, and it needs no daemon: sh -c 'trap "exit 0" TERM; sleep 30' under a 1s timeout gives status == 0, timedOut == true, succeeded == false. Only the third assertion distinguishes it from a passing probe — the old Int32 return could not express it.

Hung container exec regression. Added as a dynamic test rather than a replacement, for the reason in my earlier comment: it execs a 300s sleep under a 2s timeout and asserts the call returns at all. ~5s against a live daemon, skips via .containerDependent without one.

Mutation-checked: removing the SIGKILL escalation makes it hang rather than fail — the call never returns, which is exactly the defect it guards. The host-sleep tests stay, since they pin streamCommand's own timeout behaviour and still run where there's no daemon.

Static suite: 250 tests / 24 suites green, up from 245 / 23.


One thing I noticed while writing the teardown, unrelated to this PR and not addressed here: compose down stops containers but leaves them listed, so repeated runs accumulate stopped containers. docker compose down removes them. My test deletes explicitly to stay clean. Happy to open a separate issue or PR if that's a real divergence rather than intentional.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants