Skip to content

fork-ci: ci-demand follow-ups (cache write failure shown, BOM, NaN date, tests) #141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
scuttletoby merged 2 commits into from
Nov 15, 2026
+115 −19
Merged

fork-ci: ci-demand follow-ups (cache write failure shown, BOM, NaN date, tests) #141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9a70f86
fork-ci: ci-demand follow-ups (cache write failure shown, BOM, NaN da…
scuttletoby Nov 15, 2026
1712c64
ci-demand: a huge integer cache date is fetched again, not a crash
scuttletoby Nov 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions fork-ci/actions/ci-demand/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ For each day (a run belongs to the UTC day it was created), then a `total` row:
| runs without jobs | runs whose jobs were not read because the run stopped early (see below) |

Skipped jobs count nowhere; skipped runs count as runs but cost no request. `--json` prints the same numbers as one
object (`days.<day>`, `days.total`).
object (`days.<day>`, `days.total`), plus `incomplete`, `run_list_complete` and `cache_write_failed` (see below).

Two choices to keep in mind when comparing before and after:

Expand All @@ -39,7 +39,7 @@ Two choices to keep in mind when comparing before and after:
| Input | Meaning |
| ------------------- | ------------------------------------------------------------------------------------------------------ |
| `--repo OWNER/REPO` | the repository; or |
| `--config FILE` | a JSON file of the project with `{"ciDemand": {"repo": "OWNER/REPO"}}` (other keys are ignored) |
| `--config FILE` | a UTF-8 JSON file of the project with `{"ciDemand": {"repo": "OWNER/REPO"}}` (other keys are ignored) |
| `--days N` | the N full UTC days before today; or `--since DAY --until DAY` (both included) |
| `--cache DIR` | required: the caller's folder for saved pages (made 0700 where the OS has modes) |
| `--max-requests N` | stop after N requests (default 2000) |
Expand All @@ -56,16 +56,22 @@ Two choices to keep in mind when comparing before and after:
Server errors and network trouble are retried three times with back-off.
- **Stops half-way, runs again.** A stop prints what it has, says `INCOMPLETE` (and whether the run list was
finished) and exits 3. Run the same command again later: every page it saved is read from `--cache`, so no page is
asked for twice.
asked for twice. If the cache could not be written (a read-only folder, a full disk), the result says so (in the
first line, and `cache_write_failed` in `--json`) and a stop does not say to run again: the pages that failed were not saved, so it
would ask for them again. Fix the cache folder first.
- **Cache.** A jobs page is saved when the run and all its jobs are completed, keyed by run id and attempt (a re-run
is a new attempt, so a new page). A runs-list page is saved when every run in it is completed and the day ended over
an hour ago, and is reused for 12 hours (a later re-run changes it). Writes are a `.part-` file then a rename, so a
run killed mid-write leaves no half page under a name that is read. A file that is broken (bad JSON, a bad date, a
body that is not an object), saved for another page or in another cache format (an older or newer version of this
tool) is fetched again, never a crash. A write that fails (a full disk) is reported once and the run goes on
without saving. `.part-` files older than an hour are removed.
without saving (see above). `.part-` files older than an hour are removed.
- **Any OS.** It needs Python 3 and `gh`. Where the OS has user ids (Linux, macOS), a cache folder another user owns
is refused and the folder is made 0700, also when it existed before; on Windows these checks are skipped.
is refused and the folder is made 0700, also when it existed before; on Windows these checks are skipped. So on
Linux or macOS the cache folder must allow private permissions: one on a filesystem where chmod has no effect (FAT
or exFAT, WSL's `/mnt/c`, some network mounts) is refused (exit 2). Pick a folder on a normal disk, such as
`$XDG_CACHE_HOME/ci-demand` or `~/.cache/ci-demand`. `gh` output and the config file are read as UTF-8 whatever the
locale (a BOM is fine in the config file).
- Exit codes: 0 complete, 2 bad arguments or an API error (404, no access, network down after retries), 3 incomplete.

## General vs project
Expand Down
32 changes: 22 additions & 10 deletions fork-ci/actions/ci-demand/ci_demand.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,20 @@
X-RateLimit-Remaining falls below --reserve (default 20% of X-RateLimit-Limit, so a workflow token's 1000
an hour works too) and after --max-requests requests; on 403/429 it waits Retry-After or
the reset time when that is short (MAX_WAIT), else it stops. A stop prints what it has, marked INCOMPLETE (exit 3):
run it again with the same --cache and it goes on from the pages it already saved.
run it again with the same --cache and it goes on from the pages it already saved. A cache that could not be
written is in the result (`cache_write_failed`), and then a stop does not say to run again: the pages
that failed were not saved.

Cache (--cache, a folder the caller owns): a page is saved only when it cannot change any more: a job page when the
run and all its jobs are completed (keyed by run id and attempt, so a re-run is a new page); a run-list page when
every run in it is completed and its window ended over an hour ago, and then reused for LIST_MAX_AGE (a later re-run
changes the run). Where the OS has user ids (not Windows), a folder another user owns is refused and the folder is
made 0700. Writes are a `.part-` file then a rename. A file that is unreadable, half-written, damaged, saved for
made 0700 (a filesystem where chmod has no effect is refused). Writes are a `.part-` file then a rename. A file that is unreadable, half-written, damaged, saved for
another page or in another FORMAT is fetched again; a failed write only costs the cache; `.part-` files older than
an hour go.
Exit: 0 complete, 2 bad arguments or an API error, 3 incomplete (budget).
"""
import argparse, contextlib, datetime as dt, hashlib, json, os, re, subprocess, sys, time
import argparse, contextlib, datetime as dt, hashlib, json, math, os, re, subprocess, sys, time

MAX_WAIT = 120 # seconds: a longer rate-limit wait stops the run (INCOMPLETE) instead
LIST_MAX_AGE = 12 * 3600
Expand Down Expand Up @@ -102,8 +104,13 @@ def cached(self, key, max_age=None):
and isinstance(doc.get('body'), dict)):
return None
saved = doc.get('saved')
if isinstance(saved, bool) or not isinstance(saved, (int, float)) or saved > now() + 60:
return None # a damaged date: fetched again
try: # float(): json.load reads NaN, Infinity and any-size integers; a huge one raises OverflowError
date_ok = not isinstance(saved, bool) and isinstance(saved, (int, float)) and math.isfinite(float(saved)) \
and 0 <= saved <= now() + 60
except OverflowError:
date_ok = False
if not date_ok:
return None # a damaged date (NaN, infinite, huge, before 1970 or in the future): fetched again
if max_age is not None and now() - saved > max_age:
return None
self.hits += 1
Expand Down Expand Up @@ -367,7 +374,7 @@ def repo_of(args):
raise ValueError('give --repo or --config, one of them')
repo = args.repo
if args.config:
with open(args.config, encoding='utf-8') as f:
with open(args.config, encoding='utf-8-sig') as f: # utf-8-sig: a BOM (some Windows editors) is fine
repo = ((json.load(f) or {}).get('ciDemand') or {}).get('repo')
if not repo:
raise ValueError(f'{args.config}: no ciDemand.repo')
Expand Down Expand Up @@ -417,12 +424,17 @@ def main(argv=None):
return 2
out = report(summarize(runs, jobs))
missing = sum(r['runs_without_jobs'] for k, r in out.items() if k != 'total')
head = (f'{repo} {since}..{until} (UTC days, by run creation): {api.requests} requests, {api.hits} cached pages'
+ (f'; INCOMPLETE: {stopped}, ' + ('' if listed else 'run list not finished, ')
+ f'{missing} runs without jobs; run again with the same --cache' if stopped else ''))
head = f'{repo} {since}..{until} (UTC days, by run creation): {api.requests} requests, {api.hits} cached pages'
if stopped:
head += (f'; INCOMPLETE: {stopped}, ' + ('' if listed else 'run list not finished, ') + f'{missing} runs without jobs'
+ ('' if api.save_failed else '; run again with the same --cache'))
if api.save_failed: # nothing new was saved: a run again would fetch the same pages, never further
head += f'; cache not written ({shown(api.save_failed)})' + (
', so a run again asks for the same pages: fix --cache first' if stopped else '')
if args.json:
print(json.dumps({'repo': repo, 'since': str(since), 'until': str(until), 'requests': api.requests,
'cached': api.hits, 'incomplete': stopped, 'run_list_complete': listed, 'days': out}, indent=1))
'cached': api.hits, 'incomplete': stopped, 'run_list_complete': listed,
'cache_write_failed': api.save_failed, 'days': out}, indent=1))
else:
print(head)
print(text(out))
Expand Down
86 changes: 82 additions & 4 deletions fork-ci/actions/ci-demand/ci_demand.test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
pages (testdata/) and logs each call: no network.
Run: python3 fork-ci/actions/ci-demand/ci_demand.test.py
"""
import contextlib, copy, io, json, os, shutil, sys, tempfile, unittest
import contextlib, copy, io, json, os, shutil, subprocess, sys, tempfile, unittest
from unittest import mock

sys.dont_write_bytecode = True # leave no __pycache__ in the action folder
Expand All @@ -22,7 +22,8 @@
PAGES = json.load(f)['pages']

# The fake gh: answers from $FAKE_GH_ROUTES ({endpoint: [answer, ...]}, the n-th call gets the n-th answer, the
# last one repeats), logs argv to $FAKE_GH_LOG. An answer is {"status", "headers", "body"} or {"raw": stdout}.
# last one repeats), logs argv to $FAKE_GH_LOG. An answer is {"status", "headers", "body"} or {"raw": stdout}; a
# header set to null is left out. Like the real gh, it writes the body as UTF-8 bytes whatever the locale.
FAKE_GH = r'''
import json, os, sys
log = os.environ['FAKE_GH_LOG']
Expand All @@ -36,8 +37,9 @@
if 'raw' in a:
sys.stdout.write(a['raw']); sys.stderr.write('connection refused\n'); sys.exit(1)
h = {"X-Ratelimit-Limit": "5000", "X-Ratelimit-Remaining": "4000", "X-Ratelimit-Reset": "1793797200", **a.get('headers', {})}
body = a['body'] if isinstance(a['body'], str) else json.dumps(a['body'])
sys.stdout.write(f"HTTP/2.0 {a['status']} X\r\n" + ''.join(f"{k}: {v}\r\n" for k, v in h.items()) + "\r\n" + body)
body = a['body'] if isinstance(a['body'], str) else json.dumps(a['body'], ensure_ascii=False)
head = f"HTTP/2.0 {a['status']} X\r\n" + ''.join(f"{k}: {v}\r\n" for k, v in h.items() if v is not None)
sys.stdout.buffer.write((head + "\r\n" + body).encode('utf-8'))
sys.exit(0 if a['status'] == 200 else 1)
'''

Expand Down Expand Up @@ -114,6 +116,7 @@ def test_per_day_numbers(self):
self.assertEqual(tot['queue_wait_s']['self-hosted,build']['max'], 1200.0)
self.assertEqual(d1['never_started_jobs'], 1)
self.assertIsNone(doc['incomplete'])
self.assertIsNone(doc['cache_write_failed'])

def test_text_output(self):
code, out = self.run_main()
Expand Down Expand Up @@ -259,6 +262,8 @@ def test_stop_inside_the_run_list_says_so(self):
code, out = self.run_main('--max-requests', '1')
self.assertEqual(code, 3)
self.assertIn('INCOMPLETE: --max-requests 1 reached, run list not finished', out)
self.assertIn('run again with the same --cache', out) # the page it got was saved
self.assertNotIn('cache not written', out)

def test_cache_folder_is_private(self):
self.numbers()
Expand All @@ -268,6 +273,11 @@ def test_cache_files_damaged_or_in_another_format_are_fetched_again(self):
self.assertEqual(self.numbers()[0], 0)
damage = {'date not a number': lambda d: d.update(saved='yesterday'), 'no date': lambda d: d.pop('saved'),
'date in the future': lambda d: d.update(saved=NOW + 86400), 'body not an object': lambda d: d.update(body=[]),
'date NaN': lambda d: d.update(saved=float('nan')), # json.load reads NaN, never stale
'date -Infinity': lambda d: d.update(saved=float('-inf')),
'date a huge integer': lambda d: d.update(saved=10 ** 400), # float() of it overflows
'date a huge negative integer': lambda d: d.update(saved=-10 ** 400),
'date before 1970': lambda d: d.update(saved=-1),
'older format (none)': lambda d: d.pop('v'), 'older format': lambda d: d.update(v=M.FORMAT - 1),
'newer format': lambda d: d.update(v=M.FORMAT + 1)}
for what, change in damage.items():
Expand Down Expand Up @@ -299,6 +309,29 @@ def half(obj, f, **kw): # the disk fills up in the middle of a page
self.assertEqual(os.listdir(self.cache), []) # no half page under its final name, no .part- left
self.assertEqual(self.numbers()[1]['requests'], 6)

def test_a_cache_that_cannot_be_written_shows_in_the_result(self):
# A read-only folder or a full disk: nothing is saved, so "run again with the same --cache" would fetch the
# same pages again (and a range over --max-requests would never finish). The result says so, --json too.
def read_only(src, dst):
raise OSError(30, 'Read-only file system')

with mock.patch.object(M.os, 'replace', read_only):
code, doc = self.numbers()
self.assertEqual(code, 0, self.err)
self.assertIn('Read-only file system', doc['cache_write_failed'])
self.assertIsNone(doc['incomplete'])
code, doc = self.numbers('--max-requests', '4')
self.assertEqual(code, 3)
self.assertIn('Read-only file system', doc['cache_write_failed'])
code, out = self.run_main('--max-requests', '4')
self.assertEqual(code, 3)
head = out.splitlines()[0]
self.assertIn('INCOMPLETE: --max-requests 4 reached', head)
self.assertIn('cache not written (', head)
self.assertNotIn('run again with the same --cache', head)
self.assertIn('fix --cache first', head)
self.assertEqual(os.listdir(self.cache), [])

def test_a_write_killed_half_way_leaves_only_a_part_file(self):
class Killed(BaseException): # the process dies: nothing in the tool gets to clean up
pass
Expand Down Expand Up @@ -343,6 +376,15 @@ def test_existing_cache_folder_is_made_private(self):
self.assertEqual(self.numbers()[0], 0, self.err)
self.assertEqual(os.stat(self.cache).st_mode & 0o777, 0o700)

def test_cache_folder_that_cannot_be_made_private_is_refused(self):
# FAT, WSL /mnt/c, some network mounts: chmod "works" but the mode stays open to others.
os.makedirs(self.cache)
os.chmod(self.cache, 0o755)
with mock.patch.object(M.os, 'chmod', lambda path, mode: None):
code, doc = self.numbers()
self.assertEqual((code, doc, self.calls()), (2, None, []))
self.assertIn('cannot be made private (0700)', self.err)

def test_cache_folder_of_another_user_is_refused(self):
os.makedirs(self.cache)
other = os.stat(self.cache).st_uid + 1
Expand Down Expand Up @@ -382,6 +424,13 @@ def test_429_waits_retry_after_and_goes_on(self):
self.assertEqual((code, self.slept), (0, [7]))
self.assertEqual(doc['days']['total']['ci_hours'], 3.25)

def test_reserve_without_a_limit_header_is_1000(self):
self.routes[DAY2] = [ok(PAGES[DAY2], **{'X-Ratelimit-Limit': None, 'X-Ratelimit-Remaining': '900'})]
self.serve(self.routes)
code, doc = self.numbers()
self.assertEqual(code, 3, self.err)
self.assertIn('900 left, reserve 1000', doc['incomplete'])

def test_secondary_limit_without_retry_after_waits_a_minute(self):
self.routes[DAY1] = [{'status': 403, 'body': {'message': 'You have exceeded a secondary rate limit'}},
ok(PAGES[DAY1])]
Expand Down Expand Up @@ -472,6 +521,35 @@ def test_config_file_names_the_repo(self):
self.assertEqual(self.numbers(repo=('--config', cfg))[0], 2)
self.assertIn('no ciDemand.repo', self.err)

def test_config_with_a_utf8_bom_is_read(self):
cfg = os.path.join(self.d, 'project.json') # some Windows editors save UTF-8 with a byte-order mark
with open(cfg, 'w', encoding='utf-8-sig') as f:
json.dump({'ciDemand': {'repo': 'acme/widgets'}}, f)
code, doc = self.numbers(repo=('--config', cfg))
self.assertEqual(code, 0, self.err)
self.assertEqual(doc['repo'], 'acme/widgets')

def test_a_locale_that_is_not_utf8_reads_names_and_config_as_utf8(self):
# The script itself, in the C locale with UTF-8 mode off (ASCII): gh output and the config file are UTF-8
# whatever the locale (a Windows code page, an old server), so non-ASCII names survive and nothing is refused.
env = {**os.environ, 'LC_ALL': 'C', 'PYTHONCOERCECLOCALE': '0', 'PYTHONUTF8': '0'}
probe = subprocess.run([sys.executable, '-I', '-X', 'utf8=0', '-c', 'import locale; print(locale.getencoding())'],
env=env, capture_output=True, text=True, check=True).stdout.strip()
if probe.lower().replace('-', '') in ('utf8', 'utf8mb4'):
self.skipTest(f'the C locale is UTF-8 here ({probe})')
p = copy.deepcopy(PAGES)
p[DAY2]['workflow_runs'][0]['path'] = '.github/workflows/bäck-énd.yml'
self.serve({k: [ok(v)] for k, v in p.items()})
cfg = os.path.join(self.d, 'project.json')
with open(cfg, 'w', encoding='utf-8') as f:
json.dump({'owner': 'Zoë Ångström', 'ciDemand': {'repo': 'acme/widgets'}}, f, ensure_ascii=False)
r = subprocess.run([sys.executable, '-I', '-B', '-X', 'utf8=0', os.path.join(HERE, 'ci_demand.py'),
'--config', cfg, '--since', '2026-11-01', '--until', '2026-11-02', '--cache', self.cache,
'--json'], env=env, capture_output=True)
self.assertEqual(r.returncode, 0, r.stderr.decode('utf-8', 'replace'))
doc = json.loads(r.stdout)
self.assertEqual(doc['days']['2026-11-02']['workflows'], {'bäck-énd.yml': 1})

def test_days_means_full_utc_days_before_today(self):
self.routes[DAY1.replace('11-01T', '11-03T')] = [ok({'total_count': 0, 'workflow_runs': []})]
self.serve(self.routes)
Expand Down
Loading