Repository navigation
fix: Deduplicate restart-loop alerts and require stable recovery - #5
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A container configured with
--restart=alwaysand repeatedly exiting with a nonzero code generated alternating CRASHED / RECOVERED notifications instead of maintaining a single incident and reliably reporting RESTART-LOOP. The event listener processed eachdieimmediately, while polling treated briefrunningstates between failures as recovery.Changes
RestartCountas the sole source for restart-loop counting. The event stream still provides immediate CRASH/OOM alerts, but cannot inflate counter-based loop totals; first-poll baselines and long poll-gap safeguards prevent false positives.restart_recovery_stability_seconds, default 120 seconds). A momentary RUNNING state or healthy event no longer clears the incident.README.md, example configuration, and addtests/test_restart_loop.py.Verification
--restart=alwayscontainer exiting every 11 seconds. Before the fix: repeated CRASH / RECOVERED notifications before RESTART-LOOP.alert_channels: []; test containers were removed and the productionwatchdog:1.5.4was not modified or restarted.Scope
Core Docker container watchdog behavior only. Optional HA active-node monitoring remains in separate draft PR #4. This PR has not been deployed or merged.
Second and third review follow-ups
Final live retest — October 13, 2026