Skip to content

fix: Correlate planned Docker stops before reporting crashes - #7

Merged
rdwr-egore merged 2 commits into
mainfrom
feature/docker-planned-stop-correlation
Oct 14, 2026
Merged

rdwr-egore merged 2 commits into
mainfrom
feature/docker-planned-stop-correlation

Conversation

@rdwr-egore

@rdwr-egore rdwr-egore commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Fix misleading CRITICAL crash / OOM guidance when Docker explicitly requests a container stop, but the application ignores SIGTERM and is eventually killed with exit 137. A SIGKILL exit is not evidence of OOM.

Scope after holistic PR integration

Explicitly deferred

These were removed to keep the correctness fix focused.

Validation

  • 76 combined Python regression tests pass in the existing watchdog runtime image.
  • Real Docker Engine tests on m920q with isolated observer and no production notification channels:
    • docker stop with ignored SIGTERM -> exit 137, OOMKilled=false -> no CRASH.
    • unexpected exit code 2 -> one CRASH.
    • direct SIGKILL -> exit 137 -> one CRASH.
    • memory-cgroup OOM -> exit 137, OOMKilled=true -> one OOM only.
    • six automatic restarts -> one CRASH + one RESTART-LOOP.
  • Additional unit coverage: stop before/after die, wrong container ID, stop after the correlation window, later-run stop not hiding an earlier failure, OOM arrival order, delayed events after recovery, and name reuse.
  • Disposable test containers removed; running production watchdog not modified.
  • Full Dockerfile dependency build on m920q could not run with network disabled. GitHub CI performs the actual image build.

Important limits

A Docker stop event establishes a Docker-requested stop, not authorization. A missing or late stop event is treated conservatively as an unexpected crash. This PR does not prove that a stopped critical service should remain down.

Merge gate

Keep draft until GitHub CI (tests and Docker build) passes and compatibility with draft HA PR #4 is checked.

@rdwr-egore
rdwr-egore marked this pull request as ready for review October 14, 2026 15:16
@rdwr-egore
rdwr-egore merged commit 6459e3d into main Oct 14, 2026
1 check passed
@rdwr-egore
rdwr-egore deleted the feature/docker-planned-stop-correlation branch October 14, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant