[scuttletoby]

A small, read-only command, fork-ci/actions/ci-demand/ci_demand.py, that measures the CI demand of a repo, so a CI change has before and after numbers.

What it prints

For each UTC day in a range (--days N or --since/--until), plus a total:

• CI hours: the sum of job durations.
• Runs per workflow.
• Queue wait (job created -> started): p50, p90 and max, split by runner labels. Jobs that never started or are still queued are counted on their own.
• The cancelled and failed share of completed runs.

It prints text, or JSON with --json.

Two counting choices matter when comparing: a re-run's hours count on the day the run was created, and queue wait includes time a job waits for an environment approval.

Rate limit and state

• It only reads. Each call is a GET through gh api -X GET --include, and it never writes to GitHub.
• --max-requests caps the calls (default 2000). --reserve stops it while that much of the hourly limit is left (default 20% of the limit, so a workflow GITHUB_TOKEN with 1000 an hour also works).
• 403 and 429 answers wait for Retry-After, or for the reset when it is near, or 60 s for a secondary limit. A wait over 2 min stops the run. 5xx answers get 2/4/8 s back-off.
• Pages are cached in the --cache folder the caller gives. Where the OS has user ids, it refuses a folder someone else owns and makes the folder 0700, also when it existed before; on Windows these checks are skipped. A page is cached only when it can no longer change: the run window ended over an hour ago and every run and job in it is completed. Writes go to a temp file, then a rename, so a half-written file is never read. Cache files carry a format version.
• When it stops early (the budget, or the rate limit), it prints INCOMPLETE ... and exits 3. Run it again with the same --cache and it goes on from there; no stored page is asked for twice.
• A window with more than 1000 runs (the search cap) is split in two.

Tests

ci_demand.test.py has 36 tests. They use a fake gh on PATH that plays recorded API pages, so there is no network. They cover:

• the numbers;
• paging, including the 1000-run split;
• the state question, "What happens when this runs again, stops half-way, or meets existing/old/empty/corrupt/newer-version state?":
  • a second run uses only the cache;
  • today's run list (or one that ended under an hour ago) is not cached;
  • a jobs page with an unfinished job is not cached;
  • damaged cache files (bad date, no date, a date in the future, a body that is not an object, an older or newer format) are fetched again;
  • a write that fails half-way (disk full) leaves nothing and the run goes on; a write killed half-way leaves only a .part- file;
  • an existing open cache folder is made 0700; one of another user is refused; an OS without user ids (Windows) runs;
  • pages that are not finished are not cached;
  • a stop half-way, then a finish that asks for no page twice;
  • broken or foreign cache files;
  • leftover temp files;
  • an empty range;
• rate-limit handling, including the reserve hit on a jobs page and the default reserve with a 1000-an-hour token;
• arguments.

The test is listed in .github/fork-ci-tests.txt.

General vs project

• General: the whole tool. It takes any owner/repo as --repo, or reads ciDemand.repo from a project config file given with --config. It works on any OS with Python 3 and gh (owner and mode checks only where the OS has them). Nothing in it names a project, a host or a runner.
• Project: only the call. Each project picks its repo, range and cache folder and keeps its own results outside the fork.

🤖 Generated with Claude Code

[scuttletoby]

lens: correctness · reviewer: opus high (t3lens113) · read at: 91c22b5

covered: fork-ci/actions/ci-demand/ci_demand.py, ci_demand.test.py, testdata/two-days.json, README, and the state question: "What happens when this runs again, stops half-way, or meets existing/old/empty/corrupt/newer-version state?"

checks:

• Suite run offline (GH_TOKEN/GITHUB_TOKEN unset, empty GH_CONFIG_DIR, fake gh): 26/26 pass. The tool was not run against the real API.
• Numbers are right. Hours = job completed_at - started_at, never run times. Wait = job started_at - created_at. Jobs with no runner (cancelled while queued, or still queued) are counted apart and kept out of the waits and hours. Skipped jobs count nowhere, and skipped runs cost no request. Re-runs count per attempt (filter=all), and the docstring and README say so. Everything is UTC. A run, with all its attempts, belongs to the day it was created. The 1000-run split has no gap or overlap ([start, mid], [mid+1s, end]). Paging stops on seen >= total_count or an empty page, so no last page is lost. Nearest-rank percentile is right for n=1, 2 and 10.
• GET only: every call is gh api -X GET --include <one endpoint>, with no fields, input or method flags. The repo is checked against a regex. No token can be printed: only gh's stderr (300 chars) and the API body (200 chars) are shown.
• Rate limit: --max-requests counts retries. The reserve is checked after each answer, and that answer is still used. A 403/429 waits for Retry-After, a near reset, or 60 s. A wait over 120 s stops the run as INCOMPLETE (exit 3) and the cache stays usable. 5xx gets 2/4/8 s back-off.
• State: a run-list page is saved only when its window ended over an hour ago and every run in it is completed; it is reused for 12 h. A jobs page is saved only when the run and all its jobs are completed, keyed by attempt. Writes use a .part- file and then a rename. A truncated file or one with the wrong key is fetched again. Another repo or range uses other keys, because the key is the full endpoint. Old .part- files are removed. A stop half-way followed by a resume asks for no saved page twice (there is a test for this).
• Mutations (each one applied alone, then the suite run): these 5 were caught: counting skipped jobs, no secondary-limit detection, no MAX_WAIT stop, wait measured to completion, no runs-without-jobs count. These 5 were not caught: see the last finding below.

finding: MAJOR · ci_demand.py:70 · "Works on any OS" (PR body) is false: os.getuid does not exist on Windows. Api() raises AttributeError, which is not caught (only ValueError/OSError are), so every run ends in a traceback before the first request and does not exit with code 2. Fix: do the owner check only when hasattr(os, 'getuid'). A failing test (removes os.getuid, expects exit 0) is kept with the reviewer. · confidence: high

finding: minor · ci_demand.py:69 (README.md:37 "made 0700", docstring line 20 "a 0700 folder") · An existing cache folder is never made private: makedirs(mode=0o700, exist_ok=True) sets the mode only when it creates the folder. With a folder made earlier with mkdir (0755), pages of a private repo (branch names, actors, run titles) can be read by other local users. Fix: os.chmod(cache, 0o700) after the owner check. A failing test is kept. · confidence: high

finding: minor · ci_demand.py:86-91 · Corrupt or newer-format files are not fully handled, and there is no format version. A file whose key matches but whose saved is not a number crashes with TypeError (now() - 'x'). A body that is not a dict crashes with AttributeError later. Neither is fetched again as the docstring promises. There is no "v" field either, so a later version that stores trimmed bodies would be read silently by this one, and the other way round. Fix: check isinstance(saved, (int, float)) and isinstance(body, dict), and add 'v': 1 to the key check. A failing test is kept. · confidence: high

finding: minor · ci_demand.py:350 · The default --reserve 1000 cannot work with a workflow GITHUB_TOKEN, whose limit is 1000 per hour per repo. The first answer already shows fewer than 1000 left, so every in-workflow run stops INCOMPLETE after one request. The README says the tool can also run inside a workflow. Fix: a default relative to X-RateLimit-Limit (e.g. 20%), or a README line. · confidence: medium (it depends on in-workflow use being intended)

finding: minor (docs) · README.md "What it prints" · The two attribution choices change before/after comparisons and should be stated plainly. (1) A re-run days later adds its hours to the day the run was created, and a re-run of a run created before --since is not counted at all. (2) Queue wait includes time a job waits for an environment approval, not only time waiting for a runner. · confidence: high (they are behaviour choices, not bugs)

finding: minor (tests) · ci_demand.test.py · These 5 mutations still pass all 26 tests: (a) settled = True, which caches run-list pages of today or of the last hour, so runs created later in that window are missed for 12 h (every test window is settled); (b) removing api.check() from run_jobs, so a reserve stop during job pages is not seen; (c) a non-atomic save() that writes straight to the final file (only a truncation by hand is tested); (d) dropping the per-job completed condition from jobs-page caching; (e) removing the cache-folder owner check. Add a case with --until today, a reserve hit on a jobs page, and a save that fails half-way. · confidence: high

verdict: NOT SAFE. One major: the Windows crash breaks the stated any-OS promise, and the fix is two lines. Without it the numbers, GET-only, rate-limit and resume logic are correct, and this would be SAFE with minors for follow-up.

[scuttletoby]

not a lens: fix note

All in f753027. Each new test failed on 91c22b5, or (where the old code was right) on a mutant of the new code.

• MAJOR, no os.getuid on Windows: owner and mode checks run only where the OS has user ids. Test test_os_without_user_ids_runs (deletes os.getuid).
• Existing cache folder not private: made 0700, refused if that does not hold. test_existing_cache_folder_is_made_private.
• Damaged or other-format cache files: a v format field; a bad, missing or future date, a body that is not an object, an older or newer format are fetched again. A failed write is reported once and the run goes on. test_cache_files_damaged_or_in_another_format_are_fetched_again, test_a_write_that_fails_half_way_leaves_no_page_and_goes_on.
• Default reserve with a workflow token: 20% of X-RateLimit-Limit. test_default_reserve_follows_the_limit_so_a_workflow_token_works.
• README: re-runs count on the run's creation day, and queue wait includes approval time (also in the docstring).
• Test gaps: (a) test_run_list_of_today_or_the_last_hour_is_not_cached, (b) test_reserve_reached_on_a_jobs_page_stops_there, (c) test_a_write_killed_half_way_leaves_only_a_part_file, (d) test_jobs_page_with_an_unfinished_job_is_not_cached, (e) test_cache_folder_of_another_user_is_refused.
• Also: gh output and the config file are read as UTF-8, not the Windows code page.

36/36 pass offline. The reviewer's three kept tests pass. Format check is clean, and so are fork-ci-tests.cjs and the full fork-ci list.

🤖 Generated with Claude Code

[scuttletoby]

lens: correctness · reviewer: opus high (t3lens113q) · read at: f753027

covered: the fix round 91c22b56eb..f753027e42 (fork-ci/actions/ci-demand/ci_demand.py, ci_demand.test.py, README) and the code it calls (list_runs, run_jobs, main). The state question: "What happens when this runs again, stops half-way, or meets existing/old/empty/corrupt/newer-version state?"

checks:

• Suite run offline (GH_TOKEN/GITHUB_TOKEN unset, empty GH_CONFIG_DIR, fake gh): 36/36 pass. The three tests kept from round 1 now pass. The tool was not run against the real API.
• Round-1 MAJOR (Windows) is fixed at the root. With os.getuid removed, Api() skips the owner and mode checks and the run ends with exit 0 and the right numbers. Windows has nothing else in its path: the makedirs mode is ignored there, os.replace errors are caught by save, and --include output read in text mode still splits on \n\n. gh output, cache files and the config file are read as UTF-8, and stdout/stderr use errors='replace' only when the tool is run as a script.
• 0700 on an existing folder. A 0755 folder becomes 0700. A symlink to a folder you own makes the target 0700, because stat and chmod follow the link. A folder (or link target) owned by another user is refused before any chmod. A chmod that raises gives OSError, which means exit 2. A chmod that has no effect is refused with "cannot be made private (0700)". I checked all of these by hand.
• Cache versioning. Files with no v, an older v or a newer v, a date that is not a number, missing or in the future, or a body that is not an object are all fetched again, never a crash. A second run then uses the cache only. A newer tool's file is overwritten by this one (the path is the same), so two versions that share one folder cost re-fetches, not wrong numbers.
• A failed write. It writes a .part- file, then flush, fsync and close, and only then the rename. Any OSError, including one from close, unlinks the part file and leaves the final name either old or absent. A half page can never be read. A process killed during a write leaves only a .part- file, which is never read and is removed after an hour. Rate limit: every request still goes through --max-requests and the reserve, so a run that cannot write its cache cannot go past them. It only fetches again on the next run (see the first follow-up).
• Reserve = 20% of X-RateLimit-Limit: 5000 gives 1000 (as before), a workflow's 1000 gives 200, the unauthenticated 60 gives 12 (about 49 requests), and 0 to 4 gives 0, after which 403 handling takes over. A missing limit header falls back to 1000. An explicit --reserve 0 is honoured (is not None).
• Mutations, each applied alone with the suite run: 16 of 20 caught (direct os.getuid, no chmod, no owner check, no v check, no v written, no body-dict check, no future-date check, no date-type check, save without try, no part unlink, a warning on every failure, non-atomic save, fixed reserve, settled = True, no check() on jobs pages, no per-job completed). The 4 that survive are in the follow-ups.
• Round 1 is intact. GET only: the argv is unchanged (gh api -X GET --include <endpoint>). No token printed: the new message shows only the OSError (a cache path). The numbers, paging, the 1000-run split and resume are unchanged, and their tests pass.

follow-up (minor, an issue, not another round):

• minor · ci_demand.py:124 and :420-425 · A cache that cannot be written (a read-only folder, a full disk) shows only on stderr. The --json result has no field for it, and an INCOMPLETE head still says "run again with the same --cache", but nothing was saved. A range larger than --max-requests then never finishes: each run fetches the same pages again. Fix: add cache_write_failed to the JSON and the head, and drop the "run again" hint when it is set.
• minor (tests) · ci_demand.test.py · These 4 mutations pass all 36 tests: (a) removing the "cannot be made private" refusal (chmod that has no effect); (b) a missing X-RateLimit-Limit falling back to 0 instead of 1000; (c) dropping encoding='utf-8' from the gh subprocess; (d) the same for the config file. (c) and (d) need a non-UTF-8 locale or a mocked locale.getpreferredencoding.
• nit · ci_demand.py:370 · A config saved as UTF-8 with a BOM, which some Windows editors write, is refused with exit 2 ("Unexpected UTF-8 BOM"). Fix: encoding='utf-8-sig'.
• nit · ci_demand.py:106 · json.load accepts NaN. A run-list page whose saved is NaN passes every check, because every comparison with NaN is False, and it is reused forever. This needs hand damage, since the tool never writes NaN. Fix: math.isfinite(saved).
• nit (docs) · README "Any OS" · On Linux or macOS, a cache folder on a filesystem where chmod has no effect (FAT/exFAT, WSL /mnt/c, some network mounts) is now refused (exit 2). Round 1 accepted it. This is the right secure default; one README line would save a surprise.
  Failing tests for the first, third and fourth items are kept with the reviewer.

verdict: SAFE. The round-1 major and all five minors are fixed at the root, and new tests catch them. No new blocker or major. Five follow-ups above.