Skip to content

fix: Deduplicate restart-loop alerts and require stable recovery - #5

Merged
rdwr-egore merged 5 commits into
mainfrom
fix/restart-loop-lifecycle
Oct 14, 2026
Merged

rdwr-egore merged 5 commits into
mainfrom
fix/restart-loop-lifecycle

Conversation

@rdwr-egore

@rdwr-egore rdwr-egore commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A container configured with --restart=always and repeatedly exiting with a nonzero code generated alternating CRASHED / RECOVERED notifications instead of maintaining a single incident and reliably reporting RESTART-LOOP. The event listener processed each die immediately, while polling treated brief running states between failures as recovery.

Changes

  • Use Docker RestartCount as the sole source for restart-loop counting. The event stream still provides immediate CRASH/OOM alerts, but cannot inflate counter-based loop totals; first-poll baselines and long poll-gap safeguards prevent false positives.
  • Send a single initial CRASH alert, escalate to RESTART-LOOP when the authoritative counter reaches the configured threshold, and suppress per-cycle CRASH/OOM/unhealthy notifications until confirmed recovery.
  • Require continuous Docker uptime before RECOVERED for restarting CRASH/OOM/RESTART-LOOP incidents (new restart_recovery_stability_seconds, default 120 seconds). A momentary RUNNING state or healthy event no longer clears the incident.
  • Reset the incident's restart history after real recovery; handle clean stops, reused container names, and a changed restart policy without false loop or recovery alerts.
  • Update README.md, example configuration, and add tests/test_restart_loop.py.

Verification

  • 45 automated tests passed in an isolated Docker Python runtime, including existing watchdog regression tests.
  • Reproduced against the local Docker Engine using a disposable --restart=always container exiting every 11 seconds. Before the fix: repeated CRASH / RECOVERED notifications before RESTART-LOOP.
  • Live fixed-code retest: exactly one CRASH, one RESTART-LOOP, then one RECOVERED after stable uptime. For faster testing, the isolated observer used threshold 3 and a 20-second stability window; production defaults remain 5 restarts/10 minutes and 120 seconds of stability.
  • Live test observer had alert_channels: []; test containers were removed and the production watchdog:1.5.4 was not modified or restarted.

Scope

Core Docker container watchdog behavior only. Optional HA active-node monitoring remains in separate draft PR #4. This PR has not been deployed or merged.

Second and third review follow-ups

  • e45be40: prevent manual/auto probes from replacing an active loop; track container IDs across name reuse; make loop notifications once per incident.
  • dc18973: remove provisional event/counter reconciliation after delayed events exposed false-loop conditions. Polling alone counts restarts, with one rolling observation window; old event timestamps prevent reopening incidents already recovered.
  • Regression coverage: counter before delayed events, historical nonzero baselines, events preceding counter increments, events older than 10 minutes, polling gaps beyond the configured window, and old events arriving after confirmed recovery.
  • Tradeoff: loop escalation waits for the next poll (60-second interval by default). The restart window is approximate to poll timing; a long polling outage deliberately skips counter deltas of unknown age.

Final live retest — October 13, 2026

  • Two isolated test containers restarted three times with an 11-second failure cycle (threshold 3 and 20-second recovery stability only in the test observer).
  • Each produced exactly CRASH -> RESTART-LOOP -> RECOVERED, with no duplicate alerts.
  • Auto health-discovery used a real /proc check; manual exec probe failed six consecutive checks during the loop without replacing its incident.
  • No test-generated production notifications. Disposable containers were removed; the production watchdog remains unchanged.

@rdwr-egore
rdwr-egore merged commit e84669d into main Oct 14, 2026
1 check passed
@rdwr-egore
rdwr-egore deleted the fix/restart-loop-lifecycle branch October 14, 2026 15:59
@rdwr-egore rdwr-egore mentioned this pull request Oct 14, 2026
16 of 21 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant