diff --git a/.github/fork-ci-tests.txt b/.github/fork-ci-tests.txt index 3f2d9db50764..fbc940c4429c 100644 --- a/.github/fork-ci-tests.txt +++ b/.github/fork-ci-tests.txt @@ -23,6 +23,7 @@ .github/scripts/fork-tests-drift.test.cjs .github/scripts/fork-upstream-files-check.test.cjs .github/scripts/fork-workflow-triggers.test.cjs +fork-ci/actions/ci-demand/ci_demand.test.py fork-ci/actions/docs-scope/docs-scope.test.py fork-ci/actions/fleet-route/fleet_route.test.py fork-ci/actions/gradle-cache/gradle_cache.test.py diff --git a/fork-ci/README.md b/fork-ci/README.md index f19a5517d57b..c634bc338bc9 100644 --- a/fork-ci/README.md +++ b/fork-ci/README.md @@ -80,6 +80,10 @@ release through [`actions/pinned`](actions/pinned/README.md): a module and comma files a pin config names, checks each by git blob id, caches them and runs the checked bytes. Modules that are meant for this say which names callers may import in their README's "Python API" section (docs-scope, gradle-cache). +[`actions/ci-demand`](actions/ci-demand/README.md) is such a command only: a read-only measure of a repository's CI +per day (CI hours, runs per workflow, queue wait per runner label, cancelled and failed share), for a before/after +record of a CI change. + ## How a release is made [`.github/workflows/fork-ci-release.yml`](../.github/workflows/fork-ci-release.yml) runs once a day diff --git a/fork-ci/actions/ci-demand/README.md b/fork-ci/actions/ci-demand/README.md new file mode 100644 index 000000000000..9bf0c4182fba --- /dev/null +++ b/fork-ci/actions/ci-demand/README.md @@ -0,0 +1,74 @@ +# ci-demand + +Not a composite action: a read-only command that says how much CI a repository ran, per UTC day. Use it to record +a baseline before a CI change and to measure again after it. + +```sh +python3 -I ci_demand.py --repo OWNER/REPO --days 7 --cache "$XDG_CACHE_HOME/ci-demand" +python3 -I ci_demand.py --config .github/fleet.json --since 2026-11-01 --until 2026-11-07 --cache DIR --json +``` + +It needs `gh`, logged in with read access to the repository's Actions. Outside a workflow, run it pinned like any +fork-ci file: `pinned.py --config run ci-demand ci_demand.py ` ([pinned](../pinned/README.md)). + +## What it prints + +For each day (a run belongs to the UTC day it was created), then a `total` row: + +| Number | How it is counted | +| ------------------------ | ------------------------------------------------------------------------------------------------------------ | +| CI hours | the sum of `completed_at - started_at` of every job of every attempt; a cancelled job counts the time it ran | +| runs per workflow | runs by workflow file (`ci.yml`), not by run name | +| cancelled / failed share | of the day's completed runs (skipped runs included); failed = `failure`, `timed_out`, `startup_failure` | +| queue wait | job `created_at -> started_at`: n, p50, p90 (nearest rank), max, per runner label set (`self-hosted,build`) | +| never reached a runner | jobs with no runner: cancelled while queued (not in the wait numbers), or still queued now | +| runs without jobs | runs whose jobs were not read because the run stopped early (see below) | + +Skipped jobs count nowhere; skipped runs count as runs but cost no request. `--json` prints the same numbers as one +object (`days.`, `days.total`). + +Two choices to keep in mind when comparing before and after: + +- **Re-runs count on the day the run was created.** A re-run days later adds its hours to that first day, and a + re-run of a run created before `--since` is not counted at all. +- **Queue wait includes approval time.** A job that waits for an environment approval (a protection rule) counts that + time as queue wait, not only the time it waits for a runner. + +## Inputs + +| Input | Meaning | +| ------------------- | ------------------------------------------------------------------------------------------------------ | +| `--repo OWNER/REPO` | the repository; or | +| `--config FILE` | a JSON file of the project with `{"ciDemand": {"repo": "OWNER/REPO"}}` (other keys are ignored) | +| `--days N` | the N full UTC days before today; or `--since DAY --until DAY` (both included) | +| `--cache DIR` | required: the caller's folder for saved pages (made 0700 where the OS has modes) | +| `--max-requests N` | stop after N requests (default 2000) | +| `--reserve N` | stop once the rate limit has fewer than N left, so other tools keep working (default 20% of the limit) | + +## What it guarantees + +- **Read-only.** Every request is `gh api -X GET --include `: no field, body or method that could write. +- **Rate limit.** One runs-list query per day (a day over the API's 1000-run cap is split in halves), then one jobs + page per run that was not skipped. It stops at `--max-requests` or `--reserve`. A 403/429 waits `Retry-After`, + the reset time (when it is at most 2 minutes away) or 60 s for a secondary limit; a longer wait stops the run. + The default reserve is 20% of the token's hourly limit: 1000 of a user's 5000, 200 of a workflow `GITHUB_TOKEN`'s + 1000, so it also runs inside a workflow. + Server errors and network trouble are retried three times with back-off. +- **Stops half-way, runs again.** A stop prints what it has, says `INCOMPLETE` (and whether the run list was + finished) and exits 3. Run the same command again later: every page it saved is read from `--cache`, so no page is + asked for twice. +- **Cache.** A jobs page is saved when the run and all its jobs are completed, keyed by run id and attempt (a re-run + is a new attempt, so a new page). A runs-list page is saved when every run in it is completed and the day ended over + an hour ago, and is reused for 12 hours (a later re-run changes it). Writes are a `.part-` file then a rename, so a + run killed mid-write leaves no half page under a name that is read. A file that is broken (bad JSON, a bad date, a + body that is not an object), saved for another page or in another cache format (an older or newer version of this + tool) is fetched again, never a crash. A write that fails (a full disk) is reported once and the run goes on + without saving. `.part-` files older than an hour are removed. +- **Any OS.** It needs Python 3 and `gh`. Where the OS has user ids (Linux, macOS), a cache folder another user owns + is refused and the folder is made 0700, also when it existed before; on Windows these checks are skipped. +- Exit codes: 0 complete, 2 bad arguments or an API error (404, no access, network down after retries), 3 incomplete. + +## General vs project + +General: the command, its counting rules and its cache. Project: only the repository (an argument, or `ciDemand.repo` +in the project's own config file) and where the caller keeps the cache. diff --git a/fork-ci/actions/ci-demand/ci_demand.py b/fork-ci/actions/ci-demand/ci_demand.py new file mode 100644 index 000000000000..562967606ef4 --- /dev/null +++ b/fork-ci/actions/ci-demand/ci_demand.py @@ -0,0 +1,436 @@ +#!/usr/bin/env python3 +"""How much CI did a repository run, per day: read-only numbers from the GitHub Actions API. + +For every UTC day of a date range (a run belongs to the day it was created): CI hours (the sum of every job's +started -> completed time, all attempts), runs per workflow file, queue wait (job created -> started; p50, p90, max) +per runner label set, and the share of completed runs that ended cancelled or failed. + + python3 -I ci_demand.py (--repo OWNER/REPO | --config FILE) (--days N | --since DAY --until DAY) + --cache DIR [--max-requests N] [--reserve N] [--json] + +A re-run's attempts count on the day the run was created (a re-run of a run created before the range is not +counted), and queue wait includes time a job waits for an environment approval, not only for a runner. + +--days N is the N full UTC days before today. --config reads `ciDemand.repo` from a JSON file of the project. +It only ever sends GET requests (`gh api -X GET`): it never writes to GitHub. Rate limit: it stops before +X-RateLimit-Remaining falls below --reserve (default 20% of X-RateLimit-Limit, so a workflow token's 1000 +an hour works too) and after --max-requests requests; on 403/429 it waits Retry-After or +the reset time when that is short (MAX_WAIT), else it stops. A stop prints what it has, marked INCOMPLETE (exit 3): +run it again with the same --cache and it goes on from the pages it already saved. + +Cache (--cache, a folder the caller owns): a page is saved only when it cannot change any more: a job page when the +run and all its jobs are completed (keyed by run id and attempt, so a re-run is a new page); a run-list page when +every run in it is completed and its window ended over an hour ago, and then reused for LIST_MAX_AGE (a later re-run +changes the run). Where the OS has user ids (not Windows), a folder another user owns is refused and the folder is +made 0700. Writes are a `.part-` file then a rename. A file that is unreadable, half-written, damaged, saved for +another page or in another FORMAT is fetched again; a failed write only costs the cache; `.part-` files older than +an hour go. +Exit: 0 complete, 2 bad arguments or an API error, 3 incomplete (budget). +""" +import argparse, contextlib, datetime as dt, hashlib, json, os, re, subprocess, sys, time + +MAX_WAIT = 120 # seconds: a longer rate-limit wait stops the run (INCOMPLETE) instead +LIST_MAX_AGE = 12 * 3600 +PART_AGE = 3600 +PAGE = 100 +SEARCH_CAP = 1000 # the runs list returns at most this many runs for one `created` query +RETRIES = 3 +FORMAT = 1 # cache file format: a file of another format (older or newer tool) is fetched again +RESERVE_PCT = 20 # default --reserve: this share of X-RateLimit-Limit (1000 when the limit is not given) +sleep = time.sleep # tests replace these two +now = time.time + + +class Budget(Exception): + """The request cap or the rate-limit reserve is reached: stop and print what is there.""" + + +class ApiError(Exception): + pass + + +def shown(text): + """One printable line (names come from the API).""" + return ''.join(c if c.isprintable() else ascii(c)[1:-1] for c in str(text)) + + +def parse_include(out): + """(status, headers, body) of `gh api --include` stdout; None when it is not an HTTP answer.""" + head, sep, body = out.partition('\r\n\r\n') if '\r\n\r\n' in out else out.partition('\n\n') + lines = head.splitlines() + m = re.match(r'HTTP/\S+ (\d{3})', lines[0]) if lines else None + if not sep or not m: + return None + headers = {} + for line in lines[1:]: + k, _, v = line.partition(':') + headers[k.strip().lower()] = v.strip() + return int(m.group(1)), headers, body + + +class Api: + """GET-only `gh api` with a request cap, a rate-limit reserve, back-off and a page cache.""" + + def __init__(self, cache, max_requests, reserve, gh='gh'): + self.cache, self.max_requests, self.reserve, self.gh = cache, max_requests, reserve, gh + self.requests = self.hits = 0 + self.stop = self.save_failed = None + os.makedirs(cache, mode=0o700, exist_ok=True) # pages of a private repository: kept private + getuid = getattr(os, 'getuid', None) # None on Windows: no owner or mode bits to check there + if getuid is not None: + if os.stat(cache).st_uid != getuid(): + raise ValueError(f'--cache {cache} belongs to another user') + if os.stat(cache).st_mode & 0o077: # made earlier by someone with a wider umask + os.chmod(cache, 0o700) + if os.stat(cache).st_mode & 0o077: + raise ValueError(f'--cache {cache} cannot be made private (0700)') + for name in os.listdir(cache): # a killed write leaves a .part- file; it is never read + p = os.path.join(cache, name) + if name.startswith('.part-') and now() - os.path.getmtime(p) > PART_AGE: + os.unlink(p) + + def _path(self, key): + return os.path.join(self.cache, hashlib.sha256(key.encode()).hexdigest()[:40] + '.json') + + def cached(self, key, max_age=None): + try: + with open(self._path(key), encoding='utf-8') as f: + doc = json.load(f) + except (OSError, ValueError): + return None + if not (isinstance(doc, dict) and doc.get('v') == FORMAT and doc.get('key') == key + and isinstance(doc.get('body'), dict)): + return None + saved = doc.get('saved') + if isinstance(saved, bool) or not isinstance(saved, (int, float)) or saved > now() + 60: + return None # a damaged date: fetched again + if max_age is not None and now() - saved > max_age: + return None + self.hits += 1 + return doc['body'] + + def save(self, key, body): + """Atomic: a `.part-` file, then a rename. A failed write is reported once and only costs the cache.""" + part = os.path.join(self.cache, f'.part-{os.getpid()}-{hashlib.sha256(key.encode()).hexdigest()[:12]}') + try: + with open(part, 'w', encoding='utf-8') as f: + json.dump({'v': FORMAT, 'key': key, 'saved': now(), 'body': body}, f) + f.flush() + os.fsync(f.fileno()) + os.replace(part, self._path(key)) + except OSError as e: + with contextlib.suppress(OSError): + os.unlink(part) + if not self.save_failed: + self.save_failed = str(e) + print(f'ci-demand: cache not written ({shown(e)}); going on without it', file=sys.stderr) + + def get(self, endpoint): + """The decoded JSON of one GET; Budget when the cap or reserve would be crossed.""" + for attempt in range(RETRIES + 1): + if self.requests >= self.max_requests: + raise Budget(f'--max-requests {self.max_requests} reached') + self.requests += 1 + r = subprocess.run([self.gh, 'api', '-X', 'GET', '--include', endpoint], capture_output=True, text=True, + encoding='utf-8', errors='replace') # not the locale's code page (Windows) + got = parse_include(r.stdout) + if got is None: # network or gh trouble: no HTTP answer + if attempt == RETRIES: + raise ApiError(f'gh api {endpoint}: {shown(r.stderr.strip()[:300])}') + sleep(2 ** (attempt + 1)) + continue + status, h, body = got + remaining = int(h['x-ratelimit-remaining']) if h.get('x-ratelimit-remaining', '').isdigit() else None + if status == 200: + reserve = self.reserve_for(h) + if remaining is not None and remaining < reserve: + self.stop = f'rate limit: {remaining} left, reserve {reserve}' + try: + return json.loads(body) + except ValueError: + raise ApiError(f'{endpoint}: not JSON') + secondary = status in (403, 429) and 'rate limit' in body.lower() + if status in (403, 429) and (h.get('retry-after') or remaining == 0 or secondary): + if h.get('retry-after', '').isdigit(): + wait = int(h['retry-after']) + elif remaining == 0 and h.get('x-ratelimit-reset', '').isdigit(): + wait = int(h['x-ratelimit-reset']) - int(now()) + 1 + else: + wait = 60 # GitHub's advice for a secondary limit without Retry-After + if wait > MAX_WAIT or attempt == RETRIES: + raise Budget(f'rate limited ({status}), wait {wait}s') + sleep(max(wait, 1)) + continue + if status >= 500 and attempt < RETRIES: + sleep(2 ** (attempt + 1)) + continue + raise ApiError(f'{endpoint}: HTTP {status} {shown(body[:200])}') + raise ApiError(f'{endpoint}: gave up') # not reached + + def reserve_for(self, headers): + """--reserve, or by default RESERVE_PCT of the answer's limit (a workflow token has only 1000 an hour).""" + if self.reserve is not None: + return self.reserve + limit = headers.get('x-ratelimit-limit', '') + return int(limit) * RESERVE_PCT // 100 if limit.isdigit() else 1000 + + def check(self): + """Budget once the last answer said the reserve is reached (the answer itself was still used).""" + if self.stop: + raise Budget(self.stop) + + +def iso(t): + return t.strftime('%Y-%m-%dT%H:%M:%SZ') + + +def ts(s): + if not s: + return None + t = dt.datetime.fromisoformat(s.replace('Z', '+00:00')) + return t if t.tzinfo else t.replace(tzinfo=dt.timezone.utc) + + +def list_runs(api, repo, start, end, out): + """Every run created in [start, end] into `out` (by id). A window over the API's 1000-run cap is split in two.""" + base = f'repos/{repo}/actions/runs?created={iso(start)}..{iso(end)}&per_page={PAGE}' + settled = end < dt.datetime.fromtimestamp(now(), dt.timezone.utc) - dt.timedelta(hours=1) + page, seen = 1, 0 + while True: + ep = f'{base}&page={page}' + body = api.cached(ep, LIST_MAX_AGE) if settled else None + if body is None: + body = api.get(ep) + if settled and all(r.get('status') == 'completed' for r in body.get('workflow_runs', [])): + api.save(ep, body) + total = body.get('total_count', 0) + if total > SEARCH_CAP and end - start > dt.timedelta(seconds=1): + mid = start + (end - start) / 2 + mid = mid.replace(microsecond=0) + list_runs(api, repo, start, mid, out) + list_runs(api, repo, mid + dt.timedelta(seconds=1), end, out) + return + runs = body.get('workflow_runs', []) + for r in runs: + out[r['id']] = r + seen += len(runs) + api.check() + if not runs or seen >= total: + return + page += 1 + + +def run_jobs(api, repo, run): + """Every job of every attempt of one run.""" + jobs, page = [], 1 + while True: + ep = f'repos/{repo}/actions/runs/{run["id"]}/jobs?filter=all&per_page={PAGE}&page={page}' + key = f'{ep}#attempt={run.get("run_attempt", 1)}' + body = api.cached(key) + if body is None: + body = api.get(ep) + if run.get('status') == 'completed' and all(j.get('status') == 'completed' for j in body.get('jobs', [])): + api.save(key, body) + jobs += body.get('jobs', []) + api.check() + if not body.get('jobs') or len(jobs) >= body.get('total_count', 0): + return jobs + page += 1 + + +def pct(values, p): + """Nearest-rank percentile of a non-empty list.""" + s = sorted(values) + return s[max(0, -(-len(s) * p // 100) - 1)] + + +def label_key(job): + return ','.join(job.get('labels') or []) or '(none)' + + +def summarize(runs, jobs_by_run): + """{day: {...}} from runs and the jobs of the runs that have them (a run without jobs is counted as missing).""" + days = {} + for r in runs.values(): + day = r['created_at'][:10] + d = days.setdefault(day, {'runs': 0, 'workflows': {}, 'completed': 0, 'cancelled': 0, 'failed': 0, + 'seconds': 0, 'waits': {}, 'never_started': 0, 'still_queued': 0, + 'jobs_missing': 0}) + d['runs'] += 1 + wf = os.path.basename(r.get('path') or '') or shown(r.get('name', '?')) + d['workflows'][wf] = d['workflows'].get(wf, 0) + 1 + if r.get('status') == 'completed': + d['completed'] += 1 + d['cancelled'] += r.get('conclusion') == 'cancelled' + d['failed'] += r.get('conclusion') in ('failure', 'timed_out', 'startup_failure') + jobs = jobs_by_run.get(r['id']) + if jobs is None: + d['jobs_missing'] += r.get('conclusion') != 'skipped' + continue + for j in jobs: + if j.get('conclusion') == 'skipped': + continue + created, started, completed = ts(j.get('created_at')), ts(j.get('started_at')), ts(j.get('completed_at')) + if not j.get('runner_name'): # never reached a runner: cancelled while queued, or queued now + if j.get('status') == 'completed': + d['never_started'] += 1 + else: + d['still_queued'] += 1 + continue + if created and started: + d['waits'].setdefault(label_key(j), []).append(max(0, (started - created).total_seconds())) + if started and completed: + d['seconds'] += max(0, (completed - started).total_seconds()) + return days + + +def runs_needing_jobs(runs): + """Skipped runs ran no job: no request for them.""" + return [r for r in sorted(runs.values(), key=lambda r: r['id']) if r.get('conclusion') != 'skipped'] + + +def wait_stats(values): + return {'n': len(values), 'p50': pct(values, 50), 'p90': pct(values, 90), 'max': max(values)} + + +def report(days): + """JSON-ready numbers per day, plus 'total'.""" + out, total = {}, None + for day in sorted(days): + d = days[day] + out[day] = row(d) + if total is None: + total = {k: (dict(v) if isinstance(v, dict) else v) for k, v in d.items()} + total['waits'] = {k: list(v) for k, v in d['waits'].items()} + else: + for k, v in d.items(): + if k == 'workflows': + for w, n in v.items(): + total[k][w] = total[k].get(w, 0) + n + elif k == 'waits': + for lab, vals in v.items(): + total[k].setdefault(lab, []).extend(vals) + else: + total[k] += v + if total is not None: + out['total'] = row(total) + return out + + +def row(d): + share = lambda n: round(100 * n / d['completed'], 1) if d['completed'] else None + return {'ci_hours': round(d['seconds'] / 3600, 2), 'runs': d['runs'], 'workflows': dict(sorted(d['workflows'].items())), + 'cancelled_pct': share(d['cancelled']), 'failed_pct': share(d['failed']), + 'queue_wait_s': {lab: wait_stats(v) for lab, v in sorted(d['waits'].items())}, + 'never_started_jobs': d['never_started'], 'still_queued_jobs': d['still_queued'], + 'runs_without_jobs': d['jobs_missing']} + + +def fmt_s(s): + s = int(s) + return f'{s // 3600}h{s % 3600 // 60:02d}m' if s >= 3600 else f'{s // 60}m{s % 60:02d}s' + + +def text(out): + lines = [] + for day, r in out.items(): + lines.append(f'{day}: {r["ci_hours"]} CI hours, {r["runs"]} runs, cancelled {r["cancelled_pct"]}%, ' + f'failed {r["failed_pct"]}%' + (f', {r["runs_without_jobs"]} runs without jobs (INCOMPLETE)' + if r['runs_without_jobs'] else '')) + lines.append(' runs: ' + ', '.join(f'{shown(w)} {n}' for w, n in r['workflows'].items())) + for lab, w in r['queue_wait_s'].items(): + lines.append(f' wait {shown(lab)}: n {w["n"]}, p50 {fmt_s(w["p50"])}, p90 {fmt_s(w["p90"])}, ' + f'max {fmt_s(w["max"])}') + if r['never_started_jobs'] or r['still_queued_jobs']: + lines.append(f' jobs that never reached a runner: {r["never_started_jobs"]} (cancelled in queue), ' + f'{r["still_queued_jobs"]} still queued') + return '\n'.join(lines) + + +def day_range(args, today): + if args.days is not None: + if args.since or args.until: + raise ValueError('--days or --since/--until, not both') + if args.days < 1: + raise ValueError('--days must be at least 1') + return today - dt.timedelta(days=args.days), today - dt.timedelta(days=1) + if not (args.since and args.until): + raise ValueError('give --days N, or --since DAY and --until DAY') + since, until = (dt.date.fromisoformat(x) for x in (args.since, args.until)) + if until < since: + raise ValueError('--until is before --since') + return since, until + + +def repo_of(args): + if bool(args.repo) == bool(args.config): + raise ValueError('give --repo or --config, one of them') + repo = args.repo + if args.config: + with open(args.config, encoding='utf-8') as f: + repo = ((json.load(f) or {}).get('ciDemand') or {}).get('repo') + if not repo: + raise ValueError(f'{args.config}: no ciDemand.repo') + if not re.fullmatch(r'[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+', repo): + raise ValueError(f'not OWNER/REPO: {shown(repo)}') + return repo + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__.split('\n')[0]) + ap.add_argument('--repo') + ap.add_argument('--config') + ap.add_argument('--days', type=int) + ap.add_argument('--since') + ap.add_argument('--until') + ap.add_argument('--cache', required=True) + ap.add_argument('--max-requests', type=int, default=2000) + ap.add_argument('--reserve', type=int, help=f'leave this much of the rate limit to others ' + f'(default {RESERVE_PCT}%% of X-RateLimit-Limit)') + ap.add_argument('--json', action='store_true') + args = ap.parse_args(argv) + try: + repo = repo_of(args) + since, until = day_range(args, dt.datetime.fromtimestamp(now(), dt.timezone.utc).date()) + except (ValueError, OSError) as e: + print(f'ci-demand: {shown(e)}', file=sys.stderr) + return 2 + try: + api = Api(args.cache, args.max_requests, args.reserve) + except (ValueError, OSError) as e: + print(f'ci-demand: {shown(e)}', file=sys.stderr) + return 2 + runs, jobs, stopped, listed = {}, {}, None, False + try: + day = since + while day <= until: # one query per day keeps each window small (the 1000-run cap) and its pages cacheable + start = dt.datetime.combine(day, dt.time(), dt.timezone.utc) + list_runs(api, repo, start, start + dt.timedelta(seconds=86399), runs) + day += dt.timedelta(days=1) + listed = True + for r in runs_needing_jobs(runs): + jobs[r['id']] = run_jobs(api, repo, r) + except Budget as e: + stopped = str(e) + except ApiError as e: + print(f'ci-demand: {e}', file=sys.stderr) + return 2 + out = report(summarize(runs, jobs)) + missing = sum(r['runs_without_jobs'] for k, r in out.items() if k != 'total') + head = (f'{repo} {since}..{until} (UTC days, by run creation): {api.requests} requests, {api.hits} cached pages' + + (f'; INCOMPLETE: {stopped}, ' + ('' if listed else 'run list not finished, ') + + f'{missing} runs without jobs; run again with the same --cache' if stopped else '')) + if args.json: + print(json.dumps({'repo': repo, 'since': str(since), 'until': str(until), 'requests': api.requests, + 'cached': api.hits, 'incomplete': stopped, 'run_list_complete': listed, 'days': out}, indent=1)) + else: + print(head) + print(text(out)) + return 3 if stopped else 0 + + +if __name__ == '__main__': + for stream in (sys.stdout, sys.stderr): # a Windows console's code page cannot show every run name + if hasattr(stream, 'reconfigure'): + stream.reconfigure(errors='replace') + sys.exit(main()) diff --git a/fork-ci/actions/ci-demand/ci_demand.test.py b/fork-ci/actions/ci-demand/ci_demand.test.py new file mode 100644 index 000000000000..cdbd865dcc7a --- /dev/null +++ b/fork-ci/actions/ci-demand/ci_demand.test.py @@ -0,0 +1,494 @@ +"""ci_demand.py: per-day CI hours, runs per workflow, queue wait per runner label and cancelled/failed share, read +with GET-only `gh api` calls. Every case runs the real command against a fake `gh` on PATH that serves recorded +pages (testdata/) and logs each call: no network. +Run: python3 fork-ci/actions/ci-demand/ci_demand.test.py +""" +import contextlib, copy, io, json, os, shutil, sys, tempfile, unittest +from unittest import mock + +sys.dont_write_bytecode = True # leave no __pycache__ in the action folder + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import ci_demand as M # noqa: E402 + +R = 'repos/acme/widgets/actions/runs' +DAY1 = f'{R}?created=2026-11-01T00:00:00Z..2026-11-01T23:59:59Z&per_page=100&page=1' +DAY2 = f'{R}?created=2026-11-02T00:00:00Z..2026-11-02T23:59:59Z&per_page=100&page=1' +DAY4 = f'{R}?created=2026-11-04T00:00:00Z..2026-11-04T23:59:59Z&per_page=100&page=1' # "today" at NOW +J101, J201 = (f'{R}/{n}/jobs?filter=all&per_page=100&page=1' for n in (101, 201)) +NOW = 1793793600.0 # 2026-11-04T12:00:00Z +with open(os.path.join(HERE, 'testdata', 'two-days.json')) as f: + PAGES = json.load(f)['pages'] + +# The fake gh: answers from $FAKE_GH_ROUTES ({endpoint: [answer, ...]}, the n-th call gets the n-th answer, the +# last one repeats), logs argv to $FAKE_GH_LOG. An answer is {"status", "headers", "body"} or {"raw": stdout}. +FAKE_GH = r''' +import json, os, sys +log = os.environ['FAKE_GH_LOG'] +with open(log, 'a') as f: + f.write(json.dumps(sys.argv[1:]) + '\n') +routes = json.load(open(os.environ['FAKE_GH_ROUTES'])) +ep = sys.argv[-1] +calls = sum(1 for line in open(log) if json.loads(line)[-1] == ep) +answers = routes.get(ep) or [{"status": 404, "body": {"message": "Not Found"}}] +a = answers[min(calls, len(answers)) - 1] +if 'raw' in a: + sys.stdout.write(a['raw']); sys.stderr.write('connection refused\n'); sys.exit(1) +h = {"X-Ratelimit-Limit": "5000", "X-Ratelimit-Remaining": "4000", "X-Ratelimit-Reset": "1793797200", **a.get('headers', {})} +body = a['body'] if isinstance(a['body'], str) else json.dumps(a['body']) +sys.stdout.write(f"HTTP/2.0 {a['status']} X\r\n" + ''.join(f"{k}: {v}\r\n" for k, v in h.items()) + "\r\n" + body) +sys.exit(0 if a['status'] == 200 else 1) +''' + + +def ok(body, **headers): + return {'status': 200, 'headers': headers, 'body': body} + + +class Case(unittest.TestCase): + def setUp(self): + self.d = tempfile.mkdtemp(prefix='ci-demand-') + self.addCleanup(shutil.rmtree, self.d, True) + bin_ = os.path.join(self.d, 'bin') + os.makedirs(bin_) + with open(os.path.join(bin_, 'gh'), 'w') as f: + f.write(f'#!{sys.executable}\n{FAKE_GH}') + os.chmod(os.path.join(bin_, 'gh'), 0o755) + self.log = os.path.join(self.d, 'gh.log') + self.routes_file = os.path.join(self.d, 'routes.json') + self.cache = os.path.join(self.d, 'cache') + env = {'PATH': bin_ + os.pathsep + os.environ['PATH'], 'FAKE_GH_LOG': self.log, + 'FAKE_GH_ROUTES': self.routes_file} + old = {k: os.environ.get(k) for k in env} + os.environ.update(env) + self.addCleanup(lambda: [os.environ.pop(k) if v is None else os.environ.__setitem__(k, v) for k, v in old.items()]) + self.slept = [] + for name, val in (('now', lambda: self.clock), ('sleep', self.slept.append)): + orig = getattr(M, name) + setattr(M, name, val) + self.addCleanup(setattr, M, name, orig) + self.clock = NOW + self.serve({k: [ok(v)] for k, v in PAGES.items()}) + + def serve(self, routes): + self.routes = routes + with open(self.routes_file, 'w') as f: + json.dump(routes, f) + + def calls(self): + if not os.path.exists(self.log): + return [] + with open(self.log) as f: + return [json.loads(line) for line in f] + + def run_main(self, *args, days=('--since', '2026-11-01', '--until', '2026-11-02'), repo=('--repo', 'acme/widgets')): + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = M.main([*repo, *days, '--cache', self.cache, *args]) + self.err = err.getvalue() + return code, out.getvalue() + + def numbers(self, *args, **kw): + code, out = self.run_main('--json', *args, **kw) + return code, json.loads(out) if out else None + + +class Numbers(Case): + def test_per_day_numbers(self): + code, doc = self.numbers() + self.assertEqual(code, 0, self.err) + d1, d2, tot = doc['days']['2026-11-01'], doc['days']['2026-11-02'], doc['days']['total'] + # 30m + 5m + 10m (a cancelled job still used its runner) + 60m + 60m (both attempts of run 104) + self.assertEqual(d1['ci_hours'], 2.75) + self.assertEqual(d2['ci_hours'], 0.5) + self.assertEqual(tot['ci_hours'], 3.25) + self.assertEqual(d1['workflows'], {'ci.yml': 2, 'labels.yml': 1, 'nightly.yml': 1}) + self.assertEqual(tot['runs'], 5) + self.assertEqual((d1['cancelled_pct'], d1['failed_pct']), (25.0, 25.0)) # of 4 completed runs + self.assertEqual((d2['cancelled_pct'], d2['failed_pct']), (0.0, 0.0)) + # build waits 60, 30, 600, 120 s: nearest rank p50 60, p90 600; skipped and never-started jobs are left out + self.assertEqual(d1['queue_wait_s'], {'self-hosted,build': {'n': 4, 'p50': 60.0, 'p90': 600.0, 'max': 600.0}, + 'ubuntu-latest': {'n': 1, 'p50': 10.0, 'p90': 10.0, 'max': 10.0}}) + self.assertEqual(tot['queue_wait_s']['self-hosted,build']['n'], 5) + self.assertEqual(tot['queue_wait_s']['self-hosted,build']['max'], 1200.0) + self.assertEqual(d1['never_started_jobs'], 1) + self.assertIsNone(doc['incomplete']) + + def test_text_output(self): + code, out = self.run_main() + self.assertEqual(code, 0) + self.assertIn('2026-11-01: 2.75 CI hours, 4 runs, cancelled 25.0%, failed 25.0%', out) + self.assertIn(' wait self-hosted,build: n 4, p50 1m00s, p90 10m00s, max 10m00s', out) + self.assertIn('total: 3.25 CI hours, 5 runs', out) + self.assertNotIn('INCOMPLETE', out) + + def test_get_only_and_no_request_for_a_skipped_run(self): + self.run_main() + calls = self.calls() + self.assertTrue(calls) + for c in calls: + self.assertEqual(c[:4], ['api', '-X', 'GET', '--include'], c) + self.assertEqual(len(c), 5, c) # no -f/-F/--input/--method: nothing can be sent + self.assertFalse([c for c in calls if '/103/' in c[-1]]) # run 103 was skipped: it ran no job + self.assertEqual(len(calls), 6) + + def test_percentile_is_nearest_rank(self): + self.assertEqual(M.pct([5], 90), 5) + self.assertEqual(M.pct([1, 2, 3, 4], 50), 2) + self.assertEqual(M.pct(list(range(1, 11)), 90), 9) + self.assertEqual(M.pct(list(range(1, 11)), 91), 10) + + def test_job_still_queued_and_job_in_progress(self): + p = copy.deepcopy(PAGES) + p[DAY2]['workflow_runs'][0].update(status='in_progress', conclusion=None) + j = p[f'{R}/201/jobs?filter=all&per_page=100&page=1'] + j['jobs'][0].update(status='in_progress', completed_at=None) + j['jobs'].append({**j['jobs'][0], 'status': 'queued', 'runner_name': None, 'started_at': None}) + j['total_count'] = 2 + self.serve({k: [ok(v)] for k, v in p.items()}) + code, doc = self.numbers() + self.assertEqual(code, 0) + d2 = doc['days']['2026-11-02'] + self.assertEqual((d2['ci_hours'], d2['still_queued_jobs'], d2['cancelled_pct']), (0.0, 1, None)) + self.assertEqual(d2['queue_wait_s']['self-hosted,build']['n'], 1) # its wait is known once it started + + +class Paging(Case): + def test_runs_and_jobs_over_several_pages(self): + runs = [{'id': 300 + i, 'path': '.github/workflows/ci.yml', 'status': 'completed', 'conclusion': 'success', + 'run_attempt': 1, 'created_at': '2026-11-01T08:00:00Z'} for i in range(150)] + job = {'name': 'j', 'labels': ['x'], 'created_at': '2026-11-01T08:00:00Z', 'started_at': '2026-11-01T08:00:00Z', + 'completed_at': '2026-11-01T08:00:36Z', 'status': 'completed', 'conclusion': 'success', 'runner_name': 'r'} + routes = {DAY1: [ok({'total_count': 150, 'workflow_runs': runs[:100]})], + DAY1.replace('&page=1', '&page=2'): [ok({'total_count': 150, 'workflow_runs': runs[100:]})]} + for r in runs: + routes[f'{R}/{r["id"]}/jobs?filter=all&per_page=100&page=1'] = [ok({'total_count': 1, 'jobs': [job]})] + big = f'{R}/300/jobs?filter=all&per_page=100&page=' + routes[big + '1'] = [ok({'total_count': 120, 'jobs': [job] * 100})] + routes[big + '2'] = [ok({'total_count': 120, 'jobs': [job] * 20})] + self.serve(routes) + code, doc = self.numbers('--max-requests', '500', days=('--since', '2026-11-01', '--until', '2026-11-01')) + self.assertEqual(code, 0, self.err) + self.assertEqual(doc['days']['2026-11-01']['runs'], 150) + self.assertEqual(doc['days']['2026-11-01']['queue_wait_s']['x']['n'], 149 + 120) + self.assertEqual(doc['days']['2026-11-01']['ci_hours'], round((149 + 120) * 36 / 3600, 2)) + + def test_a_window_over_the_1000_run_cap_is_split(self): + first, second = (f'{R}?created=2026-11-01T00:00:00Z..2026-11-01T11:59:59Z&per_page=100&page=1', + f'{R}?created=2026-11-01T12:00:00Z..2026-11-01T23:59:59Z&per_page=100&page=1') + d1 = PAGES[DAY1]['workflow_runs'] + routes = {k: [ok(v)] for k, v in PAGES.items()} + routes[DAY1] = [ok({'total_count': 1500, 'workflow_runs': d1})] + routes[first] = [ok({'total_count': 3, 'workflow_runs': [r for r in d1 if r['created_at'] < '2026-11-01T12']})] + routes[second] = [ok({'total_count': 1, 'workflow_runs': [r for r in d1 if r['created_at'] >= '2026-11-01T12']})] + self.serve(routes) + code, doc = self.numbers() + self.assertEqual(code, 0, self.err) + self.assertEqual(doc['days']['2026-11-01']['runs'], 4) + self.assertIn(first, [c[-1] for c in self.calls()]) + + +class State(Case): + """Runs again, stops half-way, meets existing, old or empty state.""" + + def test_second_run_uses_the_cache_only(self): + self.assertEqual(self.numbers()[0], 0) + n = len(self.calls()) + code, doc = self.numbers() + self.assertEqual((code, len(self.calls()) - n, doc['requests'], doc['cached']), (0, 0, 0, 6)) + + def test_unfinished_pages_are_not_cached(self): + p = copy.deepcopy(PAGES) + p[DAY2]['workflow_runs'][0].update(status='in_progress', conclusion=None) + self.serve({k: [ok(v)] for k, v in p.items()}) + self.numbers() + n = len(self.calls()) + self.numbers() + again = [c[-1] for c in self.calls()[n:]] + self.assertEqual(sorted(again), sorted([DAY2, f'{R}/201/jobs?filter=all&per_page=100&page=1'])) + + def test_a_rerun_is_fetched_again_once_the_list_page_is_old(self): + self.numbers() + p = copy.deepcopy(PAGES) + p[DAY2]['workflow_runs'][0]['run_attempt'] = 2 + self.serve({k: [ok(v)] for k, v in p.items()}) + n = len(self.calls()) + self.clock = NOW + M.LIST_MAX_AGE + 1 + self.numbers() + again = sorted(c[-1] for c in self.calls()[n:]) + self.assertEqual(again, sorted([DAY1, DAY2, f'{R}/201/jobs?filter=all&per_page=100&page=1'])) + + def test_broken_or_foreign_cache_files_are_fetched_again(self): + self.numbers() + files = sorted(os.listdir(self.cache)) + with open(os.path.join(self.cache, files[0]), 'w') as f: + f.write('{"key": "trunc') # a write cut short by hand + with open(os.path.join(self.cache, files[1]), 'w') as f: + json.dump({'key': 'something else', 'body': {}}, f) # not the page it is named for + n = len(self.calls()) + code, doc = self.numbers() + self.assertEqual((code, len(self.calls()) - n), (0, 2)) + self.assertEqual(doc['days']['total']['ci_hours'], 3.25) + + def test_old_part_files_go_and_fresh_ones_stay(self): + os.makedirs(self.cache) + old, fresh = os.path.join(self.cache, '.part-1-old'), os.path.join(self.cache, '.part-2-new') + for p in (old, fresh): + open(p, 'w').close() + os.utime(old, (NOW - M.PART_AGE - 5,) * 2) + os.utime(fresh, (NOW - 10,) * 2) + self.numbers() + self.assertFalse(os.path.exists(old)) + self.assertTrue(os.path.exists(fresh)) + + def test_stop_half_way_then_run_again_finishes_without_asking_twice(self): + code, doc = self.numbers('--max-requests', '4') + self.assertEqual(code, 3) + self.assertIn('--max-requests 4', doc['incomplete']) + self.assertTrue(doc['run_list_complete']) + self.assertEqual(doc['days']['total']['runs_without_jobs'], 2) + first = [c[-1] for c in self.calls()] + code, doc = self.numbers() + self.assertEqual(code, 0) + second = [c[-1] for c in self.calls()[len(first):]] + self.assertFalse(set(first) & set(second), 'a saved page was asked for again') + self.assertEqual(doc['days']['total']['ci_hours'], 3.25) + + def test_stop_inside_the_run_list_says_so(self): + code, out = self.run_main('--max-requests', '1') + self.assertEqual(code, 3) + self.assertIn('INCOMPLETE: --max-requests 1 reached, run list not finished', out) + + def test_cache_folder_is_private(self): + self.numbers() + self.assertEqual(os.stat(self.cache).st_mode & 0o777, 0o700) + + def test_cache_files_damaged_or_in_another_format_are_fetched_again(self): + self.assertEqual(self.numbers()[0], 0) + damage = {'date not a number': lambda d: d.update(saved='yesterday'), 'no date': lambda d: d.pop('saved'), + 'date in the future': lambda d: d.update(saved=NOW + 86400), 'body not an object': lambda d: d.update(body=[]), + 'older format (none)': lambda d: d.pop('v'), 'older format': lambda d: d.update(v=M.FORMAT - 1), + 'newer format': lambda d: d.update(v=M.FORMAT + 1)} + for what, change in damage.items(): + for name in os.listdir(self.cache): + p = os.path.join(self.cache, name) + with open(p) as f: + doc = json.load(f) + change(doc) + with open(p, 'w') as f: + json.dump(doc, f) + code, doc = self.numbers() + self.assertEqual((code, doc['requests'], doc['cached']), (0, 6, 0), (what, self.err)) + self.assertEqual(doc['days']['total']['ci_hours'], 3.25, what) + self.assertEqual(self.numbers()[1]['requests'], 0) # the pages fetched again were saved again + + def test_a_write_that_fails_half_way_leaves_no_page_and_goes_on(self): + real = json.dump + + def half(obj, f, **kw): # the disk fills up in the middle of a page + f.write(json.dumps(obj)[:40]) + raise OSError(28, 'No space left on device') + + with mock.patch.object(M.json, 'dump', half): + code, doc = self.numbers() + self.assertIs(json.dump, real) + self.assertEqual(code, 0, self.err) + self.assertEqual(doc['days']['total']['ci_hours'], 3.25) + self.assertEqual(self.err.count('cache not written'), 1, self.err) + self.assertEqual(os.listdir(self.cache), []) # no half page under its final name, no .part- left + self.assertEqual(self.numbers()[1]['requests'], 6) + + def test_a_write_killed_half_way_leaves_only_a_part_file(self): + class Killed(BaseException): # the process dies: nothing in the tool gets to clean up + pass + + def half(obj, f, **kw): + f.write(json.dumps(obj)[:40]) + f.flush() + raise Killed + + with mock.patch.object(M.json, 'dump', half), self.assertRaises(Killed): + self.numbers() + left = os.listdir(self.cache) + self.assertEqual(len(left), 1) + self.assertTrue(left[0].startswith('.part-'), left) # never under the name a later run reads + code, doc = self.numbers() + self.assertEqual((code, doc['requests'], doc['days']['total']['ci_hours']), (0, 6, 3.25)) + + def test_run_list_of_today_or_the_last_hour_is_not_cached(self): + run = {'id': 401, 'path': '.github/workflows/ci.yml', 'status': 'completed', 'conclusion': 'skipped', + 'run_attempt': 1, 'created_at': '2026-11-04T01:00:00Z'} + self.serve({DAY4: [ok({'total_count': 1, 'workflow_runs': [run]})]}) + today = ('--since', '2026-11-04', '--until', '2026-11-04') + for clock, cached in ((NOW, False), (NOW, False), # today, all runs completed + (NOW + 12.5 * 3600, False), (NOW + 12.5 * 3600, False), # ended 30 min ago + (NOW + 13.5 * 3600, False), (NOW + 13.5 * 3600, True)): # ended over an hour ago + self.clock = clock + code, doc = self.numbers(days=today) + self.assertEqual((code, doc['cached'] == 1), (0, cached), clock) + + def test_jobs_page_with_an_unfinished_job_is_not_cached(self): + p = copy.deepcopy(PAGES) + p[J201]['jobs'][0].update(status='in_progress', conclusion=None, completed_at=None) # the run says completed + self.serve({k: [ok(v)] for k, v in p.items()}) + self.numbers() + n = len(self.calls()) + self.numbers() + self.assertEqual([c[-1] for c in self.calls()[n:]], [J201]) + + def test_existing_cache_folder_is_made_private(self): + os.makedirs(self.cache) + os.chmod(self.cache, 0o755) + self.assertEqual(self.numbers()[0], 0, self.err) + self.assertEqual(os.stat(self.cache).st_mode & 0o777, 0o700) + + def test_cache_folder_of_another_user_is_refused(self): + os.makedirs(self.cache) + other = os.stat(self.cache).st_uid + 1 + with mock.patch.object(os, 'getuid', lambda: other): + code, doc = self.numbers() + self.assertEqual((code, doc, self.calls()), (2, None, [])) + self.assertIn('belongs to another user', self.err) + + def test_os_without_user_ids_runs(self): + # Windows: no os.getuid and no owner or mode bits; the tool must not crash before its first request. + real = os.getuid + del os.getuid + try: + code, doc = self.numbers() + self.assertFalse(hasattr(os, 'getuid')) + finally: + os.getuid = real + self.assertEqual(code, 0, self.err) + self.assertEqual(doc['days']['total']['ci_hours'], 3.25) + + def test_empty_range_and_new_cache_folder(self): + self.serve({k: [ok({'total_count': 0, 'workflow_runs': []})] for k in (DAY1, DAY2)}) + code, out = self.run_main() + self.assertEqual(code, 0) + self.assertIn(': 2 requests, 0 cached pages', out) + self.assertTrue(os.path.isdir(self.cache)) + code, doc = self.numbers() + self.assertEqual((code, doc['days']), (0, {})) + + +class RateLimit(Case): + def test_429_waits_retry_after_and_goes_on(self): + self.routes[DAY1] = [{'status': 429, 'headers': {'Retry-After': '7'}, 'body': {'message': 'slow down'}}, + ok(PAGES[DAY1])] + self.serve(self.routes) + code, doc = self.numbers() + self.assertEqual((code, self.slept), (0, [7])) + self.assertEqual(doc['days']['total']['ci_hours'], 3.25) + + def test_secondary_limit_without_retry_after_waits_a_minute(self): + self.routes[DAY1] = [{'status': 403, 'body': {'message': 'You have exceeded a secondary rate limit'}}, + ok(PAGES[DAY1])] + self.serve(self.routes) + self.assertEqual(self.numbers()[0], 0) + self.assertEqual(self.slept, [60]) + + def test_exhausted_limit_with_a_far_reset_stops_incomplete(self): + self.routes[DAY2] = [{'status': 403, 'headers': {'X-Ratelimit-Remaining': '0', + 'X-Ratelimit-Reset': str(int(NOW) + 1800)}, + 'body': {'message': 'API rate limit exceeded'}}] + self.serve(self.routes) + code, doc = self.numbers() + self.assertEqual((code, self.slept), (3, [])) + self.assertIn('rate limited (403)', doc['incomplete']) + self.assertFalse(doc['run_list_complete']) + + def test_exhausted_limit_with_a_near_reset_waits(self): + self.routes[DAY2] = [{'status': 403, 'headers': {'X-Ratelimit-Remaining': '0', + 'X-Ratelimit-Reset': str(int(NOW) + 30)}, + 'body': {'message': 'API rate limit exceeded'}}, ok(PAGES[DAY2])] + self.serve(self.routes) + self.assertEqual(self.numbers()[0], 0) + self.assertEqual(self.slept, [31]) + + def test_reserve_stops_before_the_limit_runs_out(self): + self.routes[DAY2] = [ok(PAGES[DAY2], **{'X-Ratelimit-Remaining': '900'})] + self.serve(self.routes) + code, doc = self.numbers() + self.assertEqual(code, 3) + self.assertIn('900 left, reserve 1000', doc['incomplete']) + self.assertEqual(len(self.calls()), 2) # the page that said so is used, no request after it + self.assertEqual(doc['days']['2026-11-02']['runs'], 1) + self.assertEqual(self.numbers('--reserve', '100')[0], 0) + + def test_reserve_reached_on_a_jobs_page_stops_there(self): + self.routes[J101] = [ok(PAGES[J101], **{'X-Ratelimit-Remaining': '900'})] + self.serve(self.routes) + code, doc = self.numbers() + self.assertEqual(code, 3) + self.assertIn('900 left, reserve 1000', doc['incomplete']) + self.assertEqual([c[-1] for c in self.calls()], [DAY1, DAY2, J101]) # no request after it + self.assertEqual(doc['days']['total']['runs_without_jobs'], 4) # 101 too: its run was not finished + n = len(self.calls()) + self.assertEqual(self.numbers('--reserve', '100')[0], 0) + self.assertNotIn(J101, [c[-1] for c in self.calls()[n:]]) # the page that said stop was saved + + def test_default_reserve_follows_the_limit_so_a_workflow_token_works(self): + token = {'X-Ratelimit-Limit': '1000', 'X-Ratelimit-Remaining': '990'} # GITHUB_TOKEN: 1000 an hour + self.serve({k: [ok(v, **token)] for k, v in PAGES.items()}) + self.assertEqual(self.numbers()[0], 0, self.err) + shutil.rmtree(self.cache) + self.routes = {k: [ok(v, **token)] for k, v in PAGES.items()} + self.routes[DAY2] = [ok(PAGES[DAY2], **{**token, 'X-Ratelimit-Remaining': '150'})] + self.serve(self.routes) + code, doc = self.numbers() + self.assertEqual(code, 3) + self.assertIn('150 left, reserve 200', doc['incomplete']) + + def test_server_error_is_retried_and_a_dead_network_fails(self): + self.routes[DAY1] = [{'status': 502, 'body': 'bad gateway'}, ok(PAGES[DAY1])] + self.serve(self.routes) + self.assertEqual(self.numbers()[0], 0) + self.assertEqual(self.slept, [2]) + shutil.rmtree(self.cache) + self.serve({DAY1: [{'raw': ''}]}) + code, _ = self.numbers() + self.assertEqual(code, 2) + self.assertIn('connection refused', self.err) + self.assertEqual(len([c for c in self.calls() if c[-1] == DAY1]), 2 + M.RETRIES + 1) + + def test_not_found_or_no_access_fails_without_retry(self): + self.serve({}) + code, doc = self.numbers() + self.assertEqual((code, doc), (2, None)) + self.assertIn('HTTP 404', self.err) + self.assertEqual(len(self.calls()), 1) + + +class Arguments(Case): + def test_config_file_names_the_repo(self): + cfg = os.path.join(self.d, 'fleet.json') + with open(cfg, 'w') as f: + json.dump({'ciDemand': {'repo': 'acme/widgets'}, 'other': 1}, f) + self.assertEqual(self.numbers(repo=('--config', cfg))[0], 0) + with open(cfg, 'w') as f: + json.dump({'repo': 'acme/widgets'}, f) # a top-level repo is some other tool's + self.assertEqual(self.numbers(repo=('--config', cfg))[0], 2) + self.assertIn('no ciDemand.repo', self.err) + + def test_days_means_full_utc_days_before_today(self): + self.routes[DAY1.replace('11-01T', '11-03T')] = [ok({'total_count': 0, 'workflow_runs': []})] + self.serve(self.routes) + code, doc = self.numbers(days=('--days', '3')) + self.assertEqual((code, doc['since'], doc['until']), (0, '2026-11-01', '2026-11-03')) + + def test_bad_arguments(self): + for repo, days in ((('--repo', 'acme'), ('--days', '1')), + (('--repo', 'a/b;rm'), ('--days', '1')), + ((), ('--days', '1')), + (('--repo', 'a/b'), ('--days', '0')), + (('--repo', 'a/b'), ('--days', '1', '--since', '2026-11-01')), + (('--repo', 'a/b'), ('--since', '2026-11-02', '--until', '2026-11-01')), + (('--repo', 'a/b'), ('--since', '2026-11-02'))): + self.assertEqual(self.run_main(repo=repo, days=days)[0], 2, (repo, days)) + self.assertEqual(self.calls(), []) + + +if __name__ == '__main__': + unittest.main() diff --git a/fork-ci/actions/ci-demand/testdata/two-days.json b/fork-ci/actions/ci-demand/testdata/two-days.json new file mode 100644 index 000000000000..97dd70e27880 --- /dev/null +++ b/fork-ci/actions/ci-demand/testdata/two-days.json @@ -0,0 +1,173 @@ +{ + "_comment": "Recorded-shape pages of the GitHub Actions API (fields ci_demand.py reads, trimmed) for repo acme/widgets, 2026-11-01..02. Keys are the exact endpoints ci_demand.py requests.", + "pages": { + "repos/acme/widgets/actions/runs?created=2026-11-01T00:00:00Z..2026-11-01T23:59:59Z&per_page=100&page=1": { + "total_count": 4, + "workflow_runs": [ + { + "id": 104, + "name": "nightly", + "path": ".github/workflows/nightly.yml", + "status": "completed", + "conclusion": "failure", + "run_attempt": 2, + "created_at": "2026-11-01T02:00:00Z", + "event": "pull_request" + }, + { + "id": 103, + "name": "labels", + "path": ".github/workflows/labels.yml", + "status": "completed", + "conclusion": "skipped", + "run_attempt": 1, + "created_at": "2026-11-01T12:00:00Z", + "event": "pull_request" + }, + { + "id": 102, + "name": "ci", + "path": ".github/workflows/ci.yml", + "status": "completed", + "conclusion": "cancelled", + "run_attempt": 1, + "created_at": "2026-11-01T11:00:00Z", + "event": "pull_request" + }, + { + "id": 101, + "name": "ci", + "path": ".github/workflows/ci.yml", + "status": "completed", + "conclusion": "success", + "run_attempt": 1, + "created_at": "2026-11-01T10:00:00Z", + "event": "pull_request" + } + ] + }, + "repos/acme/widgets/actions/runs?created=2026-11-02T00:00:00Z..2026-11-02T23:59:59Z&per_page=100&page=1": { + "total_count": 1, + "workflow_runs": [ + { + "id": 201, + "name": "ci", + "path": ".github/workflows/ci.yml", + "status": "completed", + "conclusion": "success", + "run_attempt": 1, + "created_at": "2026-11-02T09:00:00Z", + "event": "pull_request" + } + ] + }, + "repos/acme/widgets/actions/runs/101/jobs?filter=all&per_page=100&page=1": { + "total_count": 3, + "jobs": [ + { + "name": "build", + "labels": ["self-hosted", "build"], + "created_at": "2026-11-01T10:00:00Z", + "started_at": "2026-11-01T10:01:00Z", + "completed_at": "2026-11-01T10:31:00Z", + "status": "completed", + "conclusion": "success", + "run_attempt": 1, + "runner_name": "runner-1" + }, + { + "name": "lint", + "labels": ["ubuntu-latest"], + "created_at": "2026-11-01T10:00:00Z", + "started_at": "2026-11-01T10:00:10Z", + "completed_at": "2026-11-01T10:05:10Z", + "status": "completed", + "conclusion": "success", + "run_attempt": 1, + "runner_name": "runner-1" + }, + { + "name": "docs", + "labels": ["self-hosted", "build"], + "created_at": "2026-11-01T10:00:00Z", + "started_at": "2026-11-01T10:00:00Z", + "completed_at": "2026-11-01T10:00:00Z", + "status": "completed", + "conclusion": "skipped", + "run_attempt": 1, + "runner_name": null + } + ] + }, + "repos/acme/widgets/actions/runs/102/jobs?filter=all&per_page=100&page=1": { + "total_count": 2, + "jobs": [ + { + "name": "build", + "labels": ["self-hosted", "build"], + "created_at": "2026-11-01T11:00:00Z", + "started_at": "2026-11-01T11:10:00Z", + "completed_at": "2026-11-01T11:20:00Z", + "status": "completed", + "conclusion": "cancelled", + "run_attempt": 1, + "runner_name": "runner-1" + }, + { + "name": "test", + "labels": ["self-hosted", "build"], + "created_at": "2026-11-01T11:00:00Z", + "started_at": "2026-11-01T11:20:00Z", + "completed_at": "2026-11-01T11:20:00Z", + "status": "completed", + "conclusion": "cancelled", + "run_attempt": 1, + "runner_name": null + } + ] + }, + "repos/acme/widgets/actions/runs/104/jobs?filter=all&per_page=100&page=1": { + "total_count": 2, + "jobs": [ + { + "name": "mutate", + "labels": ["self-hosted", "build"], + "created_at": "2026-11-01T02:00:00Z", + "started_at": "2026-11-01T02:00:30Z", + "completed_at": "2026-11-01T03:00:30Z", + "status": "completed", + "conclusion": "failure", + "run_attempt": 1, + "runner_name": "runner-1" + }, + { + "name": "mutate", + "labels": ["self-hosted", "build"], + "created_at": "2026-11-01T04:00:00Z", + "started_at": "2026-11-01T04:02:00Z", + "completed_at": "2026-11-01T05:02:00Z", + "status": "completed", + "conclusion": "failure", + "run_attempt": 2, + "runner_name": "runner-1" + } + ] + }, + "repos/acme/widgets/actions/runs/201/jobs?filter=all&per_page=100&page=1": { + "total_count": 1, + "jobs": [ + { + "name": "build", + "labels": ["self-hosted", "build"], + "created_at": "2026-11-02T09:00:00Z", + "started_at": "2026-11-02T09:20:00Z", + "completed_at": "2026-11-02T09:50:00Z", + "status": "completed", + "conclusion": "success", + "run_attempt": 1, + "runner_name": "runner-1" + } + ] + } + } +}