Repository navigation
fix: retry a watchdog alert whose publish failed - #174
Merged
Merged
Conversation
When an alert could not be published, the watchdog still closed the stale run as FAILED_FINAL, or still wrote the once-per-date marker for a missed inclusion date or a missing post-run sensor, so no later pass sent it and the alert was lost for good. A failed publish now leaves the run RUNNING with its staleAt, or writes no marker, and the next pass tries again, as the start-by check already did. A status or marker write that fails after a successful publish sends the alert again on the next pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three watchdog checks recorded an alert as sent even when publishing it failed. A stale run was still set to
FAILED_FINALwith noSFN_TIMEOUT, and a missed inclusion date or a missing post-run sensor still got its once-per-date marker with noIRREGULAR_SCHEDULE_MISSEDorPOST_RUN_SENSOR_MISSING. No later pass looked at them again, so the alert was lost for good.RUNNINGwith itsstaleAt, or no marker is written, and the next pass tries again. This is what the start-by check and the relative SLA checks already did. Now that a running trigger row has nottl, the stale row is still there for that retry.RUNNING, aRERUN_REQUESTfor that date is still refused as in progress; the date closes once a later pass publishes.