Repository navigation
Airflow processing can have "holes" of missing hours #201
Description
Sometimes when there's an issue, some hourly runs don't catch up and are lost forever. As a result, some analysis-related entries might be missing. Example: obs_* entries.
We should be able to fix this with the catch_up parameter
We should also set the start_date properly to prevent doing reprocessing since the start of time.
In particular, a good date to pick is: 2025-01-24, 15:51:00, as it's the last date of the last backfill
Example:
Activity
hellais commentedon Oct 19, 2026
Worth highlighting how we detected this because of a hole in the data for the time range of approximately:
2026-10-11 23:30and2026-10-14 07:30.The PR was landed just now, so until it's reprocessed there still will be holes and any consumer of that the from then to date should re-fetch it.