Skip to content

fix: retry a watchdog alert whose publish failed - #174

Merged
dwsmith1983 merged 1 commit into
mainfrom
fix/watchdog-publish-before-marker
Nov 22, 2026
Merged

dwsmith1983 merged 1 commit into
mainfrom
fix/watchdog-publish-before-marker

Conversation

@dwsmith1983

Copy link
Copy Markdown
Owner

Three watchdog checks recorded an alert as sent even when publishing it failed. A stale run was still set to FAILED_FINAL with no SFN_TIMEOUT, and a missed inclusion date or a missing post-run sensor still got its once-per-date marker with no IRREGULAR_SCHEDULE_MISSED or POST_RUN_SENSOR_MISSING. No later pass looked at them again, so the alert was lost for good.

  • Retry. A failed publish now logs a warning and moves on without changing anything: the stale run stays RUNNING with its staleAt, or no marker is written, and the next pass tries again. This is what the start-by check and the relative SLA checks already did. Now that a running trigger row has no ttl, the stale row is still there for that retry.
  • Duplicates over losses. A status or marker write that fails after a successful publish sends the alert again on the next pass.
  • Meanwhile. While a failed publish keeps a stale run RUNNING, a RERUN_REQUEST for that date is still refused as in progress; the date closes once a later pass publishes.
  • Docs. The watchdog, AWS architecture and alerting pages describe the retry for each of the three checks.

When an alert could not be published, the watchdog still closed the
stale run as FAILED_FINAL, or still wrote the once-per-date marker for a
missed inclusion date or a missing post-run sensor, so no later pass
sent it and the alert was lost for good.

A failed publish now leaves the run RUNNING with its staleAt, or writes
no marker, and the next pass tries again, as the start-by check already
did. A status or marker write that fails after a successful publish
sends the alert again on the next pass.
@github-actions github-actions Bot added tests Test changes lambda Lambda handlers docs Documentation labels Nov 22, 2026
@dwsmith1983 dwsmith1983 self-assigned this Nov 22, 2026
@dwsmith1983
dwsmith1983 merged commit 80e9f6c into main Nov 22, 2026
12 checks passed
@dwsmith1983
dwsmith1983 deleted the fix/watchdog-publish-before-marker branch November 22, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation lambda Lambda handlers tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant