[rdwr-egore]

Summary

Optional CyberController HA active-node monitoring using the existing local Pacemaker exporter (normally localhost:9664/metrics). No new monitoring container, firewall port, Python dependency, or Pacemaker control action. Disabled by default; reuses the current state file and alert channels.

Implementation

• Identify the promoted HAOperatorRep-Res, verify quorum and Pacemaker scrape health, and confirm role changes on two consecutive successful polls.
• The newly active node sends one HA warning; the other node records the change without sending a duplicate.
• Code-review fix (31d3f03): HA polling now has its own lightweight thread and HTTP session, independent of slow/blocked Docker polling.
• Code-review fix (31d3f03): explicitly disabling HA and restarting clears its saved baseline. Re-enabling begins with a quiet baseline; an ordinary enabled restart preserves state.
• Existing Docker failure checks, transports, and container alerts are unchanged.

Tests and actual HA validation

• 26 local unit/regression tests pass, including a deliberately blocked/broken Docker poll that cannot prevent HA polling, and disable -> role change -> re-enable.
• CI validates syntax, tests, Compose and Docker image build (see Checks tab for current head).
• User-tested manual HA switchover on October 10 from .242 to .241: .242 recognized the change and deferred alerting, and .241 logged one correct HA warning.
• Both nodes' HA exporter and watchdog were present in that test. No modifications to the running lab were made for this review.

Accepted first-release limitations

• Best-effort notification delivery: save the confirmed HA role before dispatch. Delivery failures (SMTP, Slack, syslog, SNMP) are logged, not persisted for retry; a failed state write may cause a duplicate after restart. A durable queue would be disproportionate without changing channel APIs.
• SNMP mapping: uses the existing Container Watchdog MIB, with cwFailureType=ha-failover and cwContainerName=(HA cluster). NMS integrations should classify by failure type. No new MIB/OID added.
• Planned HA service removals can still produce container CRITICAL alerts; narrowly scoped noise suppression remains a separate follow-up, not part of this PR.

Before ready to merge

• Run controlled failback .241 -> .242 and validate one reverse alert.
• Verify alert receipt at the actual destination on both nodes. The .242 SMTP host previously failed DNS resolution; locally logged alerts are not proof of email delivery.
• Confirm behavior for a supported active-node-loss scenario (no unsafe network partition with STONITH disabled).

Detailed design and QA evidence: docs/HA_ACTIVE_NODE_ALERT_DESIGN.md and docs/HA_VALIDATION.md.

DRAFT — do not merge or deploy to production yet.

October 14, 2026 holistic integration update

• Updated branch to merged main through PR fix: Correlate planned Docker stops before reporting crashes #7, including fix: Rate-limit removed container event warnings and resolve by ID #6 immutable Docker event IDs and fix: Deduplicate restart-loop alerts and require stable recovery #5 restart-loop/recovery incident handling.
• Resolved Dockerfile packaging so both ha_monitor.py and lifecycle.py are included, and composed both independent workers during startup.
• Updated the HA test stub for the lifecycle worker's initialized fields. No new production fallback path or competing alert state.
• 97/97 tests passed against the combined source in the watchdog runtime image; code compiled successfully and diff checks passed.
• The production watchdog was not modified or deployed.
• Still draft. Real reverse failback (.241 -> .242), actual destination delivery verification on both nodes, and supported active-node-loss testing are still required. HA test device was offline during this integration. Do not merge based on unit tests alone.