Conversation
Raise MAX_IN_RUN from 3 to 10, stop discarding the staleness order via random.shuffle, and stop aborting the whole run when a single preset fails. With 151 tracked locations and a 5-day staleness target, throughput needs to be ~30 locations/day; capping at 3 per run left it well under that, and a single preset failure aborted the run before any of the other selected presets got a chance to update, wasting the run's throughput entirely. Locations were already being sorted oldest-first before random.shuffle discarded that ordering right before slicing; removing the shuffle keeps the most overdue locations prioritized without needing new logic.
MAX_IN_RUN was a fixed number that would need raising by hand every time a new location gets added - the same fix would eventually be needed again. Processing the backlog until a runtime budget (4 hours, safely under the job's limit) is used instead scales automatically with backlog size. Also add a concurrency group so overlapping scheduled runs queue instead of racing to push gh-pages at the same time - a likely contributor to some of the existing workflow failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@ashkulz — I noticed my own location page (
uk) had gone stale and dug into why. Writing this up plainly with the evidence, in case it's useful.The issue
uk.ymlongh-pageshasn't updated since 2026-09-08 — 9+ days as of writing. Its actual refresh history shows a widening gap, not a one-off blip:That's structural, not a fluke. There are 151 presets (
most-active-github-users-counter --list-presets) and aSTALE_DAYS = 5target, so roughly 151 / 5 ≈ 30 locations/day need processing to keep up.MAX_IN_RUN = 3caps each run at 3, and checking the workflow's actual run history shows successful runs firing a handful of times a day (not truly hourly — GitHub's own scheduling jitter on public repos), giving a real supplied throughput well under the ~30/day required. The backlog has nowhere to go but grow.It's made worse by the failure path: on a single preset failure, the loop used to
sys.exit(1)immediately, which meant none of the other presets selected in that run got processed either — an entire run's throughput lost to one bad preset. I found 2 such failed runs in the last 20 (gh run list --workflow daily_update.yml).The fix
Three small, low-risk changes to
.github/workflows/daily_update.yml, no behavior change to the ranking logic itself:MAX_IN_RUN: 3 → 10 — closes the throughput gap against the 151-location / 5-day target with headroom to also pay down the existing backlog.random.shuffle(to_process)—to_processwas already built oldest-stale-first (via thesorted()call a few lines above), so the shuffle was actively discarding a useful ordering right before slicing. Keeping it means the most-overdue locations get priority instead of leaving it to chance.I built the binary from source (
CGO_ENABLED=0 go build, same as the workflow) and ran it standalone against live data to confirm the ranking output itself is unaffected by these changes — this only touches which/how many presets get picked each run, not how any individual preset is computed.Happy to adjust
MAX_IN_RUNto whatever number you're comfortable with — the sort-order and failure-handling fixes are the more important part; the number is just a starting point backed by the throughput math above.Why the backlog got this big
The project has grown to 151 tracked locations over time, but the per-run limit stayed fixed at 3. Every time a new location got added, more work needed doing, but capacity never grew to match — so the gap kept widening.
How bad it is right now
77 of 151 locations (over half) are already more than 5 days overdue. Under the old random-pick code, being more overdue doesn't help — a location waiting 25 days has the same odds as one waiting 6.
For a typical overdue location today: most likely picked up within about a week. Worst case, if the backlog keeps growing, it could take a month or more — there's no queue order, just chance every run.
Long-term fix
MAX_IN_RUN(raised to 10 above) would just need raising again the next time a location gets added — same problem, delayed. So this PR also replaces it with a time budget: each run keeps working through the backlog (oldest-first) until it's used about 4 hours, then stops safely. Capacity now scales automatically with backlog size, with no number to remember to bump later.Also added a concurrency guard so two scheduled runs can never overlap and race to push
gh-pagesat once, which was likely causing some of the existing failures.