Context
The monitor infers executor health from queue state, systemd state, child processes, and progress rows. That is useful, but it still has to guess whether workers are actually alive and able to claim jobs.
Proposal
Add a small executor_heartbeats or worker_heartbeats state/table recording worker_id, pid, last_seen, active job id, loop state, and recent error count.
Acceptance criteria
- Each worker periodically records liveness while idle and while running jobs.
gab monitor reports stale/missing worker heartbeats separately from process-level health.- Dashboard can show worker liveness without relying only on process tree inference.
Priority
High.
Context
The monitor infers executor health from queue state, systemd state, child processes, and progress rows. That is useful, but it still has to guess whether workers are actually alive and able to claim jobs.
Proposal
Add a small
executor_heartbeatsorworker_heartbeatsstate/table recordingworker_id, pid, last_seen, active job id, loop state, and recent error count.Acceptance criteria
gab monitorreports stale/missing worker heartbeats separately from process-level health.Priority
High.