Skip to content

fix: Rate-limit removed container event warnings and resolve by ID - #6

Merged
rdwr-egore merged 2 commits into
mainfrom
fix/event-missing-container-flood
Oct 14, 2026
Merged

rdwr-egore merged 2 commits into
mainfrom
fix/event-missing-container-flood

Conversation

@rdwr-egore

@rdwr-egore rdwr-egore commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Resolve Docker event targets by immutable container ID, with a fallback for ID-less legacy events.
  • Use the resolved container's current name for incident state and OOM / recovery handling, preventing events from a renamed instance from mutating the state of a new container using its previous name.
  • Cache missing container IDs for 60 seconds: suppress duplicate Docker inspect calls and log warnings, with hard eviction at 1,024 entries.
  • Keep the existing crash/health monitoring and alert rules unchanged.

Reproduction and validation

  • 12/12 regression tests passed in the watchdog runtime image (original 5 + 7 event tests).
  • Replaying 110 late unhealthy/die events for one removed container now results in 1 Docker lookup and 2 warnings.
  • Added controlled-clock tests for expiry and cache eviction, an ID-less event fallback test, and a rename/name-reuse incident-state test.
  • Lab 218: exercised a real removed container `die` event, replayed it 25 times, and observed 1 lookup / 1 warning. Isolated test container was removed after testing.
  • Syntax and git diff checks passed. GitHub CI is responsible for the remaining build/compose checks.

Scope and limitations

  • Production watchdog deployments were not changed.
  • The fast auto-removed-container crash-detection gap predates this PR and is not fixed here. Follow up separately with appropriate planned-operation/ephemeral-container suppression.
  • Existing poll/event unhealthy-cycle accounting is outside this PR.

@rdwr-egore
rdwr-egore merged commit 603e359 into main Oct 14, 2026
1 check passed
@rdwr-egore
rdwr-egore deleted the fix/event-missing-container-flood branch October 14, 2026 15:59
@rdwr-egore rdwr-egore mentioned this pull request Oct 14, 2026
16 of 21 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant