Skip to content

Fix growing refresh backlog in daily_update workflow - #141

Open
KayanShah wants to merge 2 commits into
ashkulz:masterfrom
KayanShah:fix/daily-update-backlog
Open

KayanShah wants to merge 2 commits into
ashkulz:masterfrom
KayanShah:fix/daily-update-backlog

Conversation

@KayanShah

@KayanShah KayanShah commented Sep 17, 2026

Copy link
Copy Markdown

@ashkulz — I noticed my own location page (uk) had gone stale and dug into why. Writing this up plainly with the evidence, in case it's useful.

The issue

uk.yml on gh-pages hasn't updated since 2026-09-08 — 9+ days as of writing. Its actual refresh history shows a widening gap, not a one-off blip:

2026-08-16 -> 2026-08-21  (+5 days)
2026-08-21 -> 2026-08-26  (+5 days)
2026-08-26 -> 2026-09-01  (+6 days)
2026-09-01 -> 2026-09-08  (+7 days)
2026-09-08 -> (nothing since, 9+ days and counting)

That's structural, not a fluke. There are 151 presets (most-active-github-users-counter --list-presets) and a STALE_DAYS = 5 target, so roughly 151 / 5 ≈ 30 locations/day need processing to keep up. MAX_IN_RUN = 3 caps each run at 3, and checking the workflow's actual run history shows successful runs firing a handful of times a day (not truly hourly — GitHub's own scheduling jitter on public repos), giving a real supplied throughput well under the ~30/day required. The backlog has nowhere to go but grow.

It's made worse by the failure path: on a single preset failure, the loop used to sys.exit(1) immediately, which meant none of the other presets selected in that run got processed either — an entire run's throughput lost to one bad preset. I found 2 such failed runs in the last 20 (gh run list --workflow daily_update.yml).

The fix

Three small, low-risk changes to .github/workflows/daily_update.yml, no behavior change to the ranking logic itself:

  1. MAX_IN_RUN: 3 → 10 — closes the throughput gap against the 151-location / 5-day target with headroom to also pay down the existing backlog.
  2. Removed random.shuffle(to_process)to_process was already built oldest-stale-first (via the sorted() call a few lines above), so the shuffle was actively discarding a useful ordering right before slicing. Keeping it means the most-overdue locations get priority instead of leaving it to chance.
  3. Stopped aborting the whole run on the first failure — a failing preset is now logged and skipped, and the rest of the batch still gets processed. Everything that succeeded is pushed at the end; the step still exits non-zero if anything failed, so it's still visible in the Actions history.

I built the binary from source (CGO_ENABLED=0 go build, same as the workflow) and ran it standalone against live data to confirm the ranking output itself is unaffected by these changes — this only touches which/how many presets get picked each run, not how any individual preset is computed.

Happy to adjust MAX_IN_RUN to whatever number you're comfortable with — the sort-order and failure-handling fixes are the more important part; the number is just a starting point backed by the throughput math above.


Why the backlog got this big

The project has grown to 151 tracked locations over time, but the per-run limit stayed fixed at 3. Every time a new location got added, more work needed doing, but capacity never grew to match — so the gap kept widening.

How bad it is right now

77 of 151 locations (over half) are already more than 5 days overdue. Under the old random-pick code, being more overdue doesn't help — a location waiting 25 days has the same odds as one waiting 6.

For a typical overdue location today: most likely picked up within about a week. Worst case, if the backlog keeps growing, it could take a month or more — there's no queue order, just chance every run.

Long-term fix

MAX_IN_RUN (raised to 10 above) would just need raising again the next time a location gets added — same problem, delayed. So this PR also replaces it with a time budget: each run keeps working through the backlog (oldest-first) until it's used about 4 hours, then stops safely. Capacity now scales automatically with backlog size, with no number to remember to bump later.

Also added a concurrency guard so two scheduled runs can never overlap and race to push gh-pages at once, which was likely causing some of the existing failures.

Raise MAX_IN_RUN from 3 to 10, stop discarding the staleness order via
random.shuffle, and stop aborting the whole run when a single preset
fails.

With 151 tracked locations and a 5-day staleness target, throughput
needs to be ~30 locations/day; capping at 3 per run left it well under
that, and a single preset failure aborted the run before any of the
other selected presets got a chance to update, wasting the run's
throughput entirely. Locations were already being sorted oldest-first
before random.shuffle discarded that ordering right before slicing;
removing the shuffle keeps the most overdue locations prioritized
without needing new logic.
MAX_IN_RUN was a fixed number that would need raising by hand every
time a new location gets added - the same fix would eventually be
needed again. Processing the backlog until a runtime budget (4 hours,
safely under the job's limit) is used instead scales automatically
with backlog size.

Also add a concurrency group so overlapping scheduled runs queue
instead of racing to push gh-pages at the same time - a likely
contributor to some of the existing workflow failures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant