feat: Optional HA active-node change alert (draft)

DEPLOYMENT.md

@@ -410,6 +410,41 @@ docker exec docker-container-watchdog tail -f /var/log/watchdog/watchdog.log
 
 ---
 
+### Optional HA role-change verification
+
+This check is only for **two-node CyberController HA** installations and is
+disabled by default. On **each HA host**:
+
+1. Verify the local exporter is active: curl -fsS http://127.0.0.1:9664/metrics.
+2. Confirm metrics for Pacemaker resources, quorum, collector scrape success,
+   and local node identity are present. Port 9664 is the verified lab setting;
+   read /etc/ha_cluster_exporter.yaml if different.
+3. Enable ha_monitor.enabled: true in watchdog-config.yaml on **both**
+   nodes; restart only the watchdog service using the normal documented
+   configuration-change procedure.
+4. Verify the baseline is stored under ha_monitor.active_node in the persistent
+   watchdog state.json; startup should send **no** HA event.
+5. During an approved HA maintenance window, perform one controlled switchover
+   and failback using the supported CyberController operations procedure.
+   Confirm exactly one notification from the newly active node for each
+   transition; check docker logs docker-container-watchdog on both hosts.
+6. Disable by setting ha_monitor.enabled: false and restarting the watchdog.
+   This clears its saved HA baseline; re-enabling starts with a fresh, quiet
+   baseline (no notification for transitions that occurred while disabled).
+   An ordinary restart while HA remains enabled keeps the confirmed baseline.
+   No HA service or Pacemaker configuration changes are necessary.
+
+Delivery is best-effort. Test the actual alert receiver on **both** nodes:
+an HA alert in watchdog logs does not prove a syslog or SMTP destination
+received it. This release has no persisted notification retry queue.
+
+Do not attempt a network-partition simulation on a cluster with STONITH
+disabled: split-brain and database inconsistency are possible. The native
+Prometheus port mismatch (9002 vs exporter 9664) is a separate appliance
+configuration issue; it does not block the watchdog's direct localhost check.
+
+---
+
 ## 5. Rollback
 
 ### 5.1 Roll Back a Failed Deployment
@@ -825,9 +860,15 @@ Configuration for the SNMP Traps channel is in [2.4.4 SNMP Traps](#244-snmp-trap
 | `1.3.6.1.4.1.89.110.1.1.0` | `cwHost` | Hostname of the alerting node |
 | `1.3.6.1.4.1.89.110.1.2.0` | `cwSummary` | Human-readable alert summary |
 | `1.3.6.1.4.1.89.110.1.3.0` | `cwContainerName` | Container name |
-| `1.3.6.1.4.1.89.110.1.4.0` | `cwFailureType` | `crashed` / `oom` / `unhealthy` / `restart-loop` / `recovered` |
+| `1.3.6.1.4.1.89.110.1.4.0` | `cwFailureType` | `crashed` / `oom` / `unhealthy` / `restart-loop` / `recovered` / `ha-failover` |
 | `1.3.6.1.4.1.89.110.1.5.0` | `cwProbeDetail` | Probe failure detail |
 
+**HA trap mapping:** this release uses the existing Container Watchdog MIB.
+For HA, cwContainerName is (HA cluster), cwFailureType is ha-failover, and the
+summary/detail fields contain previous and new active nodes. NMS rules should
+match cwFailureType instead of treating all traps as container failures. No
+new trap OID is introduced.
+
 **SNMPv3 notes:**
 
 - **Engine ID:** On the first run the watchdog logs an auto-stable engine ID derived from the hostname (look for `SNMP engine ID: 0x...` in the logs). Copy it into `v3_local_engine_id` and register it in `snmptrapd.conf` with `createUser -e 0x<id> watchdog-user SHA "..." AES "..."`. This keeps the ID stable across container restarts.

Dockerfile

@@ -11,7 +11,7 @@ RUN pip install --no-cache-dir \
     "cryptography>=42.0"
 
 # Copy agent
-COPY watchdog.py lifecycle.py .
+COPY watchdog.py lifecycle.py ha_monitor.py ./
 
 # UID 1000 account for install.sh's non-root hardening path (docker-compose.yaml
 # defaults to root; install.sh instead sets WATCHDOG_UID/GID to run as this user).

README.md

@@ -69,6 +69,65 @@ Planned-stop correlation does not implement an availability policy: long-running
 
 ---
 
+### Optional HA active-node change alert
+
+On CyberController HA installations, the watchdog can optionally alert when the
+**promoted HA Operator resource** moves from one cluster node to another.
+This is an additional application event check; it does not replace or modify
+existing Docker container health checks.
+
+Enable the following block on **both** HA nodes, after verifying that each
+node has a local Pacemaker HA cluster exporter:
+
+~~~yaml
+ha_monitor:
+  enabled: true
+  exporter_url: http://127.0.0.1:9664/metrics
+~~~
+
+The watchdog reads HA cluster resource metrics and validates local-node identity,
+cluster quorum, and Pacemaker scrape success. The first valid reading
+establishes a baseline without alerting. A new active node must be confirmed
+on two consecutive successful polls (usually 60 seconds apart). Only the
+newly active node sends one **WARNING: CyberController HA active node changed**,
+using the already configured Syslog, SMTP, Slack, and/or SNMP channels.
+Failback is handled the same way; HA state persists in the normal state.json.
+HA polling runs in its own lightweight thread with a separate HTTP session, so
+slow or unavailable Docker API probes do not block HA detection.
+
+An ordinary watchdog restart with HA enabled retains its baseline. If HA
+monitoring is **disabled and the watchdog restarted**, the old HA baseline is
+cleared; re-enabling and restarting establishes a new baseline silently rather
+than reporting an unobserved, potentially old role change.
+
+**Delivery is best-effort:** after confirming a change, the watchdog saves its
+new baseline before dispatching. If SMTP/Slack/SNMP/syslog delivery fails,
+this event is logged but is **not retried** by the HA monitor. Validate the
+notification path on both HA nodes. A failed state-file save may result in a
+duplicate alert after a restart.
+
+**SNMP compatibility:** HA events use the existing Container Watchdog MIB with
+cwFailureType=`ha-failover`, cwContainerName=`(HA cluster)`, and old/new
+node values in cwSummary/cwProbeDetail. NMS integrations should distinguish
+HA events by cwFailureType, not by the container-name field.
+
+This check is **off by default**. It does not install an HA exporter, open a
+firewall port, execute Pacemaker commands, or modify cluster roles. Both
+watchdogs/exporters must be running and able to send notifications for detection
+of a failure of the original active node. If a node is newly installed after
+the failover, the first reading is a quiet baseline and **cannot reconstruct
+the earlier event**. Loss of quorum or exporter connectivity is logged locally,
+not misreported as a completed failover. Observed time may be later than the
+actual Pacemaker transition. A confirmed promotion does not guarantee all
+CyberController services have recovered.
+
+The exporter may use port **9664** even when the built-in CyberController
+Prometheus configuration references **9002**. This optional check reads the
+exporter directly and makes **no changes** to native Prometheus configuration.
+See [design, test plan and release gates](docs/HA_ACTIVE_NODE_ALERT_DESIGN.md).
+
+---
+
 ## 2. System Overview
 
 ### Project Structure