Repository navigation
fix: judge a missing post-run sensor by whether it changed since completion - #176
Merged
Merged
Conversation
…letion POST_RUN_SENSOR_MISSING decided arrival from the sensor's updatedAt and read it only as epoch milliseconds, so an RFC3339 or epoch-seconds timestamp, or none, raised the alert although the sensor had arrived. It also ignored per-period keys and skipped any date without a baseline row, so it never fired when the sensor did not exist at completion. A post-run sensor has now arrived when it exists and either differs from its entry in the completion-time baseline or had no entry. The current sensor is put through the same store encoding as the baseline before the comparison, so set-typed values compare as the baseline holds them. Per-period keys are matched the way the baseline capture matches them, and a date with no baseline row is checked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
POST_RUN_SENSOR_MISSINGdecided whether a post-run sensor had arrived from itsupdatedAt, read only as epoch milliseconds. A sensor with an RFC3339updatedAt(the format the docs use), epoch seconds, or none raised the alert although it had arrived. The check also ignored per-period keys such asrow-count#2026-03-10, and skipped any date with no baseline row, so it never fired when the sensor did not exist at completion, the case the alert is for.A sensor record is one row per run date, so a change to it after the run is the signal and when it changed does not matter.
updatedAtcounts like any other field; the time of the write is not otherwise used.setwritten by boto3) never compared equal and the alert never fired. The current sensor is now put through the same encoding before the comparison. If that fails, a warning names the pipeline and sensor key and the sensor counts as arrived.<key>#YYYY-MM-DDand<key>#YYYYMMDDare matched the way the baseline capture matches them, and a date with no baseline row is checked.Accepted limits, stated in the docs where they affect users:
BASELINE_CAPTURE_FAILED) has no entry and counts as arrived.row-count#<date>stands in for a plainrow-counteven when the plain one changed; with both date forms present, which one is used is not fixed.