diff --git a/fork-ci/actions/pr-queue/README.md b/fork-ci/actions/pr-queue/README.md index 64aeaf05dd97c..35b13f9f2eac6 100644 --- a/fork-ci/actions/pr-queue/README.md +++ b/fork-ci/actions/pr-queue/README.md @@ -5,16 +5,6 @@ up to date with their base branch. After each push to the base branch it brings exactly one waiting pull request up to date, and it turns auto-merge back on only for a head that holds exactly the change the review read. -This folder is built in four stacked pull requests: - -1. the review rules and the config (`lens.py`, `config.py`); -2. the queue itself (`pr_queue.py`, section "The queue"); -3. armed pull requests that cannot merge (a red required check, or one that - ended without passing or never reported) are told once and passed over; -4. the "red only because the base was red" rule, the reusable workflow - `.github/workflows/fork-pr-queue.yml`, `action.yml`, `tested-base/` and the - usage sections of this file: this part. - ## Config One JSON file in the project's repository. Unknown keys, a key written twice, a @@ -93,6 +83,12 @@ heading near the top, or two `covered:`/`checks:` lines. A comment that is no report says so on its first line, `not a lens: ...`. That clears it unless one of its lines is a lens header or a `lens:` field with a sha. +A comment that looks like a report but is not one valid read is a lens of its +own, keyed by that comment. Posting the report again the right way adds a new +read and leaves the old comment blocking, and hiding it is not enough either. +The ways out: delete it, edit its first line to `not a lens: ...` with no lens +header line left in it, or post an override for the head. + ## The queue ``` @@ -109,26 +105,38 @@ first, and acts on the first one it can: the way out. A pull request in a merge state the queue neither updates nor waits for (a -conflict, `DIRTY`) is passed over, and the run says so. +conflict, `DIRTY`) is passed over, and the run says so. `UNKNOWN` (GitHub is +still computing it, often for a few seconds after a push to the base) is listed +again every `poll_seconds`, for at most `wait_seconds`, before it is passed +over, so a younger pull request is not updated ahead of it. An armed pull request that cannot merge does not hold the queue. It is told why on the pull request, once per head, check and reason, and passed over: - a required check whose newest run failed (`FAILURE`, `ERROR`, `TIMED_OUT`, - or `CANCELLED` after a red with nothing green since), behind or not. Its - author fixes it; nothing in the queue re-runs it. + or `CANCELLED` after a red with nothing green since), behind or not. The + newest run of the name counts, whichever workflow ran it, as the ruleset + reads it. Its author fixes it; nothing in the queue re-runs it. - up to date, and a required check's newest run ended without passing and - without a red (`CANCELLED`, `STARTUP_FAILURE`, `STALE`, ...): nothing + without a red (`CANCELLED`, `STARTUP_FAILURE`, `STALE`, ...), and no + workflow runs on the head any more (a newer run may still be queued): nothing re-runs it by itself. The comment says to re-run the CI run or push. - up to date, and a required check never reported although no workflow runs - on the head any more. The comment says to push or merge the base: a re-run - repeats the jobs of the workflow files the head was built with, so it cannot - add a check those files do not have (one added or renamed since). + on the head any more, also when no check at all reported. The comment says + to push or merge the base: a re-run repeats the jobs of the workflow files + the head was built with, so it cannot add a check those files do not have + (one added or renamed since). + +A head with no check and no workflow run at all still holds the queue: nothing +tells "not started yet" from "never starts" (every workflow filtered out by +its paths) without a clock, and the queue keeps no state for one. Only the newest run of a check counts, and a check not in `required_checks` is ignored. Only the queue's own comment (posted with its token) counts as told: -a copy of it by anyone else does not silence it. When GitHub cannot say which workflows still run on the head, the -pull request is not called stuck: it holds the queue, and the run says why. +a copy of it by anyone else does not silence it. A pull request whose head +moved between the list and the comment is told by the next run, on its new +head. When GitHub cannot say which workflows still run on the head, the pull +request is not called stuck: it holds the queue, and the run says why. `gh` must be logged in for the repository (`GH_TOKEN`). Git fetches into the queue's own bare repository (`--cache`) and gets its login from @@ -140,8 +148,9 @@ other repository's job, as a user of its own. The `--state-file` and `--cache` folders belong to that user alone. git runs with no hooks, no fsmonitor and no replace refs, and a cache that is not that user's alone (mode `0700`), has a config the queue did not write, or holds a file that changes what its objects -say (`info/grafts`, `shallow`, `objects/info/alternates` or -`objects/info/http-alternates`) is rebuilt, never used. A job of the same user +say or how a merge reads them (`info/grafts`, `shallow`, +`objects/info/alternates`, `objects/info/http-alternates` or +`info/attributes`) is rebuilt, never used. A job of the same user can still change the cache between that check and git's next read: only the runner rule above stops that. @@ -169,7 +178,15 @@ was: give each queue its own `--state-file`. A plan without `repo` (the source queue's format) is this repository's. An unreadable plan, or one another user could have written (not this user's, or writable by its group or others), is kept as `STATE.bad`, the run fails, and every later run warns while that file -exists. +exists. A plan does not say who turned auto-merge off: a pull request a person +disarmed after the stop is re-armed too, if its head holds the change read. + +The plan is on disk, and that is why the runner rule above exists. It could +live in a comment of the queue's own login instead (trusted by +`viewerDidAuthor`, as the told notices are): the next run reads the review and +the base from GitHub anyway, and a failed write before the disarm would only +mean no disarm. That stays open for a queue that moves several pull requests at +once, and for GitHub-hosted runners. `--lock-file` (default: `STATE.lock`) keeps two runs from overlapping. `--dry-run` takes no lock and writes nothing: no `STATE`, no cache, no comment. @@ -191,7 +208,10 @@ red required checks is therefore updated when, for every red required check: When the base tip is not green yet, the pull request is told once, per head and check, that it failed as on its base and that the queue updates it by -itself once the base is green. It does not hold the queue meanwhile. When +itself once the base is green. The notice names the first check still red on +the base; when the base fixes that one and another is still red there, a new +notice names that one. A red of the pull request's own beside a base red is +named as its own ("fix the cause"). It does not hold the queue meanwhile. When anything is unknown (no note among the job's first 100 annotations, a failed read, more than 100 checks on a commit), the red counts as the pull request's own, as without the rule. There is no state and no loop: after the update the new runs test a @@ -300,12 +320,14 @@ replaced, and the replaced corpus was checked against the source copy before it was committed. A rule change here must keep the corpus green or say in its pull request why a recorded answer changes. -The port differs on purpose in three ways, and the replay checks each recorded +The port differs on purpose in four ways, and the replay checks each recorded answer that changes for its reason. The source's repository is private, so its comments carry no author association: the replay gives them `OWNER` and unedited. A read shorter than 40 hex, and a lens comment edited after posting, read nothing here (54 recorded answers). The source's `full_sha`, which matched -a short read to the pull request's commits, is gone. +a short read to the pull request's commits, is gone. A hidden (minimized) lens +comment reads nothing: the source never read `isMinimized`, and no recorded +call has it, so no recorded answer changes for it. `pr_queue.py` is a port of the same project's queue, and `pr_queue.test.py` holds that project's queue tests with the port's names. What the port changed: @@ -317,7 +339,12 @@ queue's own advice in a comment keeps its backticks and `` placeholder required check that never reported, the comment says to push or merge the base; the source said to re-run, which cannot add a check the head's workflow files do not have. With two red checks the comment names the first by name -(the source named whichever the check list gave first). +(the source named whichever the check list gave first). A red check is the +newest run of its name whichever workflow ran it, and a red that a green re-run +and then a cancelled run followed is not red (the source took the newest run +per workflow, and counted any red before a cancel). An `UNKNOWN` merge state is +listed again (the source passed it over at once), and a cancelled check waits +while a workflow still runs on the head (the source called it stuck at once). The "base was red" rule is a port of a change proposed to the source queue, with its tests. What the port changed: the rule is off unless the config turns diff --git a/fork-ci/actions/pr-queue/lens.py b/fork-ci/actions/pr-queue/lens.py index 4e2448b718e13..f30c949d6388c 100644 --- a/fork-ci/actions/pr-queue/lens.py +++ b/fork-ci/actions/pr-queue/lens.py @@ -27,10 +27,17 @@ # The name ends at the first field: ` · reviewer:` or ` · read at:` (also `read at `; the separator may also be # `,` `—` or none). A bare `reviewer` or `read at` stays in the name (`data-integrity reviewer 1` and `... 2` are two # lenses). -FIELD_RE = re.compile(r'\s*(?:[·,—]\s*)?\b(?:reviewer\s*:|read[\s-]*at\b\s*(?::|[0-9a-f]{7}))', re.I) -READ_AT_RE = re.compile(r'\bread[\s-]*at\b\s*:?\s*([0-9a-f]{7,40})\b', re.I) +# Every pattern here runs in time linear in the line, also for a trusted author's 64 KB comment: a blank run is tried +# once, from its start ((?`, until nothing changes) + # copied the whole name per pass: minutes for a 64 KB name of 13 K markers. The same steps on a word list, where a + # word stripped empty at an end stays as the blank in front of a marker until the next pass, as in the text. + words = name.lower().split() + lo, hi = 0, len(words) + while True: + while lo < hi and not words[lo]: + lo += 1 + while lo < hi and not words[hi - 1]: + hi -= 1 + if lo == hi: + return 'unnamed' + before = (words[lo], words[hi - 1], hi) + words[lo] = words[lo].lstrip('*_`') + words[hi - 1] = words[hi - 1].rstrip('*_`') + if hi - lo >= 2 and words[hi - 1] and re.fullmatch(r'r\d+|lens|re-?check', words[hi - 1]): + hi -= 1 + if (words[lo], words[hi - 1], hi) == before: + return ' '.join(w for w in words[lo:hi] if w) or 'unnamed' def names_a_lens(name): @@ -266,11 +286,12 @@ def lens_posts(self, pr): def lens_entries(self, pr): """{lens key: ((when, index), sha or None, why or None)}: each lens's NEWEST header, from the posts that count (counts). A comment that looks like a lens report but is not one valid read (lens_problem) is an entry of its - own with no sha: it read nothing until it is re-posted as one comment per lens with a first-line header, or an - override covers it. A lens comment edited after posting (includesCreatedEdit not false: GitHub records that it - changed, not what) or hidden (minimized) read nothing until that lens posts a new comment. A lens-looking post - that does not count is logged (log only, never a comment), so a wrong lens_author_associations shows in the - run.""" + own, keyed by the comment, with no sha: a correct new post of that lens does not replace it, and hiding it + does not clear it. It reads nothing until it is deleted, its first line is edited to `not a lens: ...` with no + lens header line left in it, or a newer override covers it (NO_HEADER_WAY_OUT). A lens comment edited after + posting (includesCreatedEdit not false: GitHub records that it changed, not what) or hidden (minimized) reads + nothing until that lens posts a new comment. A lens-looking post that does not count is logged (log only, never + a comment), so a wrong lens_author_associations shows in the run.""" found = {} for when, i, post in self.lens_posts(pr): where = post.get('url') or f'by {(post.get("author") or {}).get("login") or "?"} at {when or "?"}' diff --git a/fork-ci/actions/pr-queue/lens.test.py b/fork-ci/actions/pr-queue/lens.test.py index 164dbb4656f93..6520689975a29 100644 --- a/fork-ci/actions/pr-queue/lens.test.py +++ b/fork-ci/actions/pr-queue/lens.test.py @@ -208,6 +208,41 @@ def test_a_strangers_huge_comment_costs_no_time(self): # SEC-2 r3 major: anyon self.head(self.OWN, who) self.assertLess(time.monotonic() - start, 5, (body[:20], who.get('authorAssociation'))) + def test_a_trusted_authors_long_blank_or_marker_runs_cost_no_time(self): # r4 sec: 30-60 s each before + import time + shapes = ('lens: a' + ' ' * 65536 + 'b', # FIELD_RE: one blank run per start + f'lens: a · read at: {SHA_A}\nread at' + ' ' * 65536 + 'x', # READ_AT_RE: \\s*:?\\s* split a run + f'lens: a · read at: {SHA_A}\nread at' + '\u00a0' * 65536 + 'x', + 'lens: a' + ' r1' * 21000, 'lens: a' + ' lens' * 13000) # lens_key: one pass per marker + for body in shapes: + start = time.monotonic() + self.head(post(body)) + self.assertLess(time.monotonic() - start, 5, body[:20]) + + def test_many_capped_stranger_posts_cost_no_time(self): # r4 sec: 0.76 s per 4096-char post before + import time + body = 'lens: a' + ' ' * (lens.STRANGER_SCAN - 9) + 'b' + posts = [stranger(body, url=f'https://github.test/pr/7#c{i}') for i in range(20)] + start = time.monotonic() + self.assertEqual(self.head(self.OWN, *posts), (True, SHA_A, 'every lens read it')) + self.assertLess(time.monotonic() - start, 5) + + def test_a_malformed_lens_comment_blocks_until_deleted_edited_or_overridden(self): + # r3: the advice "re-post it" could not work: the malformed comment is its own entry, keyed by the comment + fence = '`' * 3 + bad = post(f'{fence}\nlens: correctness · reviewer: x · read at: {SHA_A}\n{fence}', url='https://github.test/c1') + good = post(f'lens: correctness · reviewer: x · read at: {SHA_A}', when='2026-01-02T00:00:00Z') + self.assertEqual(self.head(bad, good)[:2], (True, None), 'a correct re-post leaves the old one in force') + self.assertEqual(self.head(dict(bad, isMinimized=True), good)[:2], (True, None), 'hiding is not enough') + self.assertEqual(self.head(good), (True, SHA_A, 'every lens read it'), 'deleted') + edited = dict(bad, body='not a lens: an old draft, see the next comment', includesCreatedEdit=True) + self.assertEqual(self.head(edited, good), (True, SHA_A, 'every lens read it'), 'its first line edited') + kept = dict(bad, body=f'not a lens: draft\nlens: correctness · reviewer: x · read at: {SHA_A}') + self.assertEqual(self.head(kept, good)[:2], (True, None), 'the opt-out does not clear a header line left in it') + over = post(f'tool ready --head {SHA_A}: read it', when='2026-01-03T00:00:00Z') + self.assertEqual(self.head(bad, good, over), (True, SHA_A, 'every lens read it'), 'a newer override') + self.assertIn('delete that comment (hiding it is not enough)', lens.NO_HEADER_WAY_OUT) + def test_a_lens_comment_whose_author_does_not_count_is_logged(self): # a wrong lens_author_associations makes reviewed PRs look unreviewed: the run log must say why lines = [] @@ -252,6 +287,55 @@ def test_an_override_counts_whether_or_not_its_association_writes_lenses(self): self.assertEqual(r.reviewed_head(pr(lens_post, over)), (True, SHA_B, 'every lens read it')) +class Reviews(unittest.TestCase): + """PR reviews take part like comments (lens_posts): ordered by submittedAt, by the same author rules. gh gives no + isMinimized for a review, so a review counts as shown.""" + R = WhoCounts.R + + def review(self, body, when='2026-01-02T00:00:00Z', **extra): + p = post(body, when=..., **extra) + p['submittedAt'] = when + return p + + def head(self, comments=(), reviews=()): + return self.R.reviewed_head({'headRefOid': SHA_B, 'comments': list(comments), 'reviews': list(reviews)}) + + def test_a_counted_review_is_a_read(self): + self.assertEqual(self.head(reviews=[self.review(f'lens: correctness · reviewer: x · read at: {SHA_B}')]), + (True, SHA_B, 'every lens read it')) + + def test_the_newest_of_comments_and_reviews_counts_by_submitted_at(self): + old = post(f'lens: correctness · reviewer: x · read at: {SHA_A}', when='2026-01-01T00:00:00Z') + newer = self.review(f'lens: correctness · reviewer: x · read at: {SHA_B}', when='2026-01-02T00:00:00Z') + self.assertEqual(self.head([old], [newer]), (True, SHA_B, 'every lens read it')) + older = self.review(f'lens: correctness · reviewer: x · read at: {SHA_B}', when='2025-12-31T00:00:00Z') + self.assertEqual(self.head([old], [older]), (True, SHA_A, 'every lens read it')) + + def test_a_review_without_a_hidden_state_counts_as_shown(self): + r = self.review(f'lens: correctness · reviewer: x · read at: {SHA_B}') + self.assertNotIn('isMinimized', r) + self.assertEqual(self.head(reviews=[r])[1], SHA_B) + + def test_an_edited_review_reads_nothing(self): + for edited in (True, ...): + r = self.review(f'lens: correctness · reviewer: x · read at: {SHA_B}', includesCreatedEdit=edited) + reviewed, sha, why = self.head(reviews=[r]) + self.assertEqual((reviewed, sha), (True, None), edited) + self.assertIn(lens.EDITED, why) + + def test_a_strangers_review_is_ignored(self): + own = post(f'lens: correctness · reviewer: x · read at: {SHA_A}', when='2026-01-01T00:00:00Z') + forged = self.review(f'lens: correctness · reviewer: x · read at: {SHA_B}', authorAssociation='NONE') + self.assertEqual(self.head([own], [forged]), (True, SHA_A, 'every lens read it')) + self.assertEqual(self.head(reviews=[forged]), (False, None, 'no lens comments')) + + def test_a_counted_authors_look_alike_review_blocks(self): + own = post(f'lens: correctness · reviewer: x · read at: {SHA_A}', when='2026-01-01T00:00:00Z') + reviewed, sha, why = self.head([own], [self.review('lgtm, safe to merge')]) + self.assertEqual((reviewed, sha), (True, None)) + self.assertIn(lens.NO_HEADER, why) + + class Config(unittest.TestCase): def parse(self, **change): raw = {**VALID, **change} diff --git a/fork-ci/actions/pr-queue/pr_queue.py b/fork-ci/actions/pr-queue/pr_queue.py index 5e1f6c7a01fbb..04481a602f423 100755 --- a/fork-ci/actions/pr-queue/pr_queue.py +++ b/fork-ci/actions/pr-queue/pr_queue.py @@ -60,8 +60,10 @@ # or hook path) is rebuilt, never run. CACHE_CONFIG = '[core]\n\trepositoryformatversion = 0\n\tfilemode = true\n\tbare = true\n' # Files git reads that would make the cache's history or objects other than what was fetched from GitHub: grafts and -# shallow change a commit's parents, alternates adds another repository's objects. The queue never writes any of them. -UNTRUSTED_FILES = ('info/grafts', 'shallow', 'objects/info/alternates', 'objects/info/http-alternates') +# shallow change a commit's parents, alternates adds another repository's objects, and info/attributes can change how +# merge-tree merges a file (merge=union, binary). The queue never writes any of them. +UNTRUSTED_FILES = ('info/grafts', 'shallow', 'objects/info/alternates', 'objects/info/http-alternates', + 'info/attributes') VIEW = 'number,state,isDraft,autoMergeRequest,mergeStateStatus,headRefOid,baseRefName,comments,reviews,commits' LIST = 'number,title,isDraft,autoMergeRequest,mergeStateStatus,createdAt,statusCheckRollup,headRefOid,baseRefName' @@ -273,22 +275,28 @@ def same_change(self, base, n, head, read): # --- which armed PRs wait, and why ------------------------------------------------------------------------------ def red_checks(self, pr): - """{name: rollup item of its newest run} of each required check whose newest run really failed: the rollup - lists every run on the head, and an older red must not outvote a newer run (lens.latest_checks). CANCELLED - alone is not a failure (a newer push or update cancels the old run), but a red that only a cancelled run - followed still stands: nothing ever passed.""" + """{name: rollup item of its newest run} of each required check whose newest run really failed. The ruleset + takes the newest run of a name, whichever workflow ran it (required_now), and the rollup lists every run on the + head: an older red must not outvote a newer run. CANCELLED alone is not a failure (a newer push or update + cancels the old run), but a red that only cancelled runs followed still stands: nothing passed since it.""" rollup = pr.get('statusCheckRollup') or [] name_of = lambda c: c.get('name') or c.get('context') state_of = lambda c: (c.get('conclusion') or c.get('state') or '').upper() - out = {} + when = lambda c: c.get('startedAt') or '9999' + newest = {} for c in lens.latest_checks(rollup): name = name_of(c) - if name not in self.cfg.required_checks: - continue - if state_of(c) in RED_STATES or (state_of(c) == 'CANCELLED' and - any(name_of(o) == name and state_of(o) in RED_STATES for o in rollup)): - if (c.get('startedAt') or '') >= ((out.get(name) or {}).get('startedAt') or ''): + if name in self.cfg.required_checks and (name not in newest or when(c) >= when(newest[name])): + newest[name] = c + out = {} + for name, c in newest.items(): + if state_of(c) == 'CANCELLED': + runs = [o for o in rollup if name_of(o) == name] + green = max((when(o) for o in runs if lens.check_state(o) == 'ok'), default='') + if any(state_of(o) in RED_STATES and when(o) > green for o in runs): out[name] = c + elif state_of(c) in RED_STATES: + out[name] = c return out def failing(self, pr): @@ -322,25 +330,31 @@ def workflow_runs(self, head): def stuck(self, p): """An up-to-date armed PR that cannot merge and that nothing moves on any more: (check, what) for a required check whose newest run ended without passing and without a red (CANCELLED, STARTUP_FAILURE, STALE...), or that - never reported although no workflow runs on the head any more. None: it may still merge (green or running).""" - now = self.required_now(p.get('statusCheckRollup') or []) + never reported, in both cases only while no workflow runs on the head any more (a newer run queued with no job + yet may still report it). None: it may still merge (green or running), or GitHub could not say (printed). A + head with no check and no workflow run at all waits: nothing tells "not started yet" from "never starts" + (README.md, "The queue").""" + rollup = p.get('statusCheckRollup') or [] + now = self.required_now(rollup) ended = sorted((name, state) for name, state in now.items() if state not in ('ok', 'running')) - if ended: - return ended[0][0], f'ended {ended[0][1]}' missing = sorted(self.cfg.required_checks - set(now)) - if 'running' in now.values() or not missing: - return None - if not (p.get('statusCheckRollup') or []): - print(f'#{p["number"]}: no check has started on its head yet') + if 'running' in now.values() or not (ended or missing): return None head = (p.get('headRefOid') or '').lower() runs = self.workflow_runs(head) if runs is None: # unknown is not "nothing runs": wait, and say it - print(f'#{p["number"]}: required check {", ".join(missing)} has not reported, and which workflows still ' - f'run on {head[:12] or "its head"} is unknown: waiting') + what = (f'required check {ended[0][0]} ended {ended[0][1]}' if ended + else f'required check {", ".join(missing)} has not reported') + print(f'#{p["number"]}: {what}, and which workflows still run on {head[:12] or "its head"} is unknown: ' + 'waiting') return None if any((x.get('status') or '').lower() != 'completed' for x in runs): return None + if ended: + return ended[0][0], f'ended {ended[0][1]}' + if not rollup and not runs: + print(f'#{p["number"]}: no check and no workflow run on its head yet: waiting') + return None return missing[0], NEVER def stuck_fix(self, what, base): @@ -350,14 +364,18 @@ def stuck_fix(self, what, base): 'it, so that the current workflows run') return 'Nothing re-runs it by itself: re-run the CI run (Actions, "Re-run failed jobs") or push' - def tell_red(self, n, check, what='is red', fix='Fix the cause and push'): + def tell_red(self, n, head, check, what='is red', fix='Fix the cause and push'): """An armed PR the queue skips says why on the PR, once per head, check and reason. No answer from GitHub: the next run tries again. Told means a comment of the queue's own login (viewerDidAuthor) that starts with the mark: - anyone can post a copy of it on a public repository.""" + anyone can post a copy of it on a public repository. `head` is the head the reason was found on (the list): + after a push the reason is not known for the new head, and a mark for it would silence its real notice.""" pr = self.view(n) if not pr: print(f'#{n}: could not read it to say why it waits') return + if (pr.get('headRefOid') or '').lower() != (head or '').lower(): + print(f'#{n}: its head moved from {(head or "?")[:12]} since the list: the next run tells it') + return mark = f'required check {check} {what} on {(pr.get("headRefOid") or "?")[:12]}' told = f'pr_queue: {mark},' if any(c.get('viewerDidAuthor') is True and (c.get('body') or '').startswith(told) @@ -430,35 +448,37 @@ def commit_checks(self, expr, cache): return out def red_from_old_base(self, p, cache): - """A BEHIND PR whose required checks failed only because the base they were tested on was red. (None, None) - when, for EVERY red required check C, the base commit its run really merged with (tested_base) has a real + """A BEHIND PR whose required checks failed only because the base they were tested on was red. (None, None, + False) when, for EVERY red required check C, the base commit its run really merged with (tested_base) has a real failure of C (BASE_RED) and the base branch has C green now: an update makes a new merge on that green tip. - Otherwise (why not, base_check): base_check names a check that failed on the base too and is still not green - there, set only when EVERY red check failed on its tested base (one red of the PR's own, or one nothing proves, - means its author fixes it: False), so the PR waits for its base, not for its author. No state and - no loop: after the update the next run names a base where C is green, so a red then is the PR's own and it is - passed over (told); a re-run of the old merge names the old red base again and is updated, which it needs.""" + Otherwise (why not, check, base_red): `check` is the check that decided, for the notice (None: no one check + did). base_red is True only when EVERY red check failed on its tested base and `check` failed on the base too + and is still not green there, so the PR waits for its base, not for its author; one red of the PR's own, or one + nothing proves, means its author fixes it, and `check` names that one. No state and no loop: after the update + the next run names a base where C is green, so a red then is the PR's own and it is passed over (told); a re-run + of the old merge names the old red base again and is updated, which it needs.""" reds, base = self.red_checks(p), p.get('baseRefName') or '' if not re.fullmatch(r'[\w./-]+', base): - return 'its base branch is unknown', False + return 'its base branch is unknown', None, False tip = self.commit_checks(f'refs/heads/{base}', cache) if not tip: - return f'the checks of {base} are unknown (see above)', False + return f'the checks of {base} are unknown (see above)', None, False waits = None for check, c in sorted(reds.items()): tested = self.tested_base(c) if not tested: - return f'the base its {check} run tested is unknown (see above)', False + return f'the base its {check} run tested is unknown (see above)', check, False then = self.commit_checks(tested, cache) if not then: - return f'the checks of {base} commit {tested[:12]} are unknown (see above)', False + return f'the checks of {base} commit {tested[:12]} are unknown (see above)', check, False was = then[1].get(check, 'not reported') if was not in BASE_RED: # running, ok, cancelled, skipped, stale...: nothing proves the base was red - return f'{check} was not red on the {base} commit {tested[:12]} its run tested ({was})', False + return f'{check} was not red on the {base} commit {tested[:12]} its run tested ({was})', check, False now = tip[1].get(check, 'not reported') if now != 'ok' and not waits: # keep looking: a later red of the PR's own is its author's to fix - waits = f'{check} failed on {base} {tested[:12]} too and is not green on {base} now ({now})', check - return waits or (None, None) + waits = (f'{check} failed on {base} {tested[:12]} too and is not green on {base} now ({now})', check, + True) + return waits or (None, None, False) # --- steps 2-4 -------------------------------------------------------------------------------------------------- def view(self, n): @@ -495,8 +515,9 @@ def read_advice(self, why): """The way out when the review names no head, for the queue's comment.""" if lens.NO_HEADER in why: return ('A comment that looks like a lens report counts only with a lens header as its first plain line, ' - f'not in a code block ({self.cfg.lens_format_doc}): re-post it that way, or someone who read the ' - f'head posts the override `{self.cfg.override_prefix} : `.') + f'not in a code block ({self.cfg.lens_format_doc}). Posting it again that way does not clear ' + f'the old one: {lens.NO_HEADER_WAY_OUT} someone who read the head posts the override ' + f'`{self.cfg.override_prefix} : `.') if lens.SHORT in why: return f'A lens read names the full 40-hex sha ({self.cfg.lens_format_doc}): re-post it with that.' if lens.HIDDEN in why: @@ -697,6 +718,19 @@ def dry_run(self): return self.run(True) def run(self, dry): + """One sweep in age order: update or wait for the first PR that can move. An UNKNOWN merge state (GitHub is + still computing it, often for seconds after a push to the base) is listed again every poll_seconds, at most + wait_seconds, before the PR is passed over, so a younger PR is not updated ahead of it.""" + waits = self.cfg.wait_seconds // self.cfg.poll_seconds + while True: + code = self.sweep(dry, waits > 0) + if code is not None: + return code + waits -= 1 + self.sleep(self.cfg.poll_seconds) + + def sweep(self, dry, may_wait): + """run()'s one look at the list; None: list again (an UNKNOWN merge state, only when may_wait).""" limit = self.cfg.list_limit r = self.gh('pr', 'list', '--repo', self.repo, '--state', 'open', '--limit', str(limit), '--json', LIST) try: @@ -714,29 +748,34 @@ def run(self, dry): n = p['number'] if p.get('isDraft') or not p.get('autoMergeRequest'): continue - bad = self.failing(p) + bad, head = self.failing(p), p.get('headRefOid') or '' if bad and p.get('mergeStateStatus') == 'BEHIND' and self.cfg.update_when_base_was_red: base = p.get('baseRefName') or 'its base' - why, base_check = self.red_from_old_base(p, cache) + why, check, base_red = self.red_from_old_base(p, cache) if why is None: reds = ', '.join(sorted(self.red_checks(p))) print(f'update #{n} {(p.get("title") or "")[:70]}: required check {reds} failed on a {base} ' f'commit where it failed too, and it is green on {base} now') return self.update(n, dry) + bad = check or bad # the notice names the check that decided, and is keyed by it print(f'skip #{n}: required check {bad} failed and an update would not fix it yet: {why}') - if base_check: + if base_red: if not dry: - self.tell_red(n, bad, f'failed as on {base}', - f'{base} failed {base_check} too, so this is not this PR\'s fault: the queue ' - f'updates it by itself once {base_check} is green on {base}') + self.tell_red(n, head, bad, f'failed as on {base}', + f'{base} failed {bad} too, so this is not this PR\'s fault: the queue ' + f'updates it by itself once {bad} is green on {base}') continue elif bad: print(f'skip #{n}: required check {bad} failed (its author fixes it)') if bad: if not dry: - self.tell_red(n, bad) + self.tell_red(n, head, bad) continue if p.get('mergeStateStatus') != 'BEHIND': + if p.get('mergeStateStatus') == 'UNKNOWN' and may_wait: + print(f'#{n}: merge state UNKNOWN (GitHub is still computing it): listing again in ' + f'{self.cfg.poll_seconds} s') + return None if p.get('mergeStateStatus') not in WAITS_FOR: print(f'skip #{n}: merge state {p.get("mergeStateStatus")}, not one the queue updates or waits for') continue @@ -745,7 +784,8 @@ def run(self, dry): print(f'skip #{n}: up to date, but required check {why[0]} {why[1]} and nothing re-runs it: it ' 'cannot merge, so it does not hold the queue') if not dry: - self.tell_red(n, why[0], why[1], self.stuck_fix(why[1], p.get('baseRefName') or 'its base')) + self.tell_red(n, head, why[0], why[1], + self.stuck_fix(why[1], p.get('baseRefName') or 'its base')) continue print(f'#{n} is up to date and waiting for its checks: queue is moving') return 0 diff --git a/fork-ci/actions/pr-queue/pr_queue.test.py b/fork-ci/actions/pr-queue/pr_queue.test.py index 10c3ea9851762..200cfa30c65db 100644 --- a/fork-ci/actions/pr-queue/pr_queue.test.py +++ b/fork-ci/actions/pr-queue/pr_queue.test.py @@ -267,7 +267,7 @@ def assertOff(self, f, code, said): self.assertNotRegex(f.comments[0], r'(?i)read at\s*:', 'a queue comment must never count as a lens read') self.assertIn(said, f.comments[0]) self.assertIn('`tool ready 7`', f.comments[0]) - post = {'body': f.comments[0], 'author': {'login': 'owner'}} + post = note(f.comments[0], 't') # an author whose lens comments count: a look-alike would block (NO_HEADER) self.assertIsNone(REVIEW.lens_problem(post)) self.assertEqual(REVIEW.reviewed_head({'comments': [post], 'reviews': []}), (False, None, 'no lens comments')) @@ -367,10 +367,18 @@ def test_a_headerless_lens_comment_says_how_to_repost(self): f = self.fake(comments=[note(f'{fence}\n{LENS}: correctness · reviewer: x · read at: {self.c[:7]}\n{fence}', 't')]) code, out = self.update(f) self.assertOff(f, code, 'unclear') - self.assertIn('re-post it', f.comments[0]) + self.assertIn('Posting it again that way does not clear the old one', f.comments[0]) + self.assertIn('delete that comment (hiding it is not enough)', f.comments[0]) + self.assertIn('edit it to start with `not a lens` and a colon', f.comments[0]) self.assertIn('docs/review.md', f.comments[0]) self.assertIn('`tool ready --head : `', f.comments[0]) + def test_a_headerless_lens_comment_re_posted_right_still_blocks(self): # what the advice says, kept true + fence = '`' * 3 + bad = note(f'{fence}\n{LENS}: correctness · reviewer: x · read at: {self.c[:7]}\n{fence}', 't1') + f = self.fake(comments=[bad, lens_at('correctness', self.c, 't2')]) + self.assertOff(f, self.update(f)[0], 'unclear') + def test_failed_update_or_arm_stays_off(self): f = self.fake(fail=('update',)) self.assertOff(f, self.update(f)[0], 'update-branch failed') @@ -755,6 +763,40 @@ def test_the_plan_is_this_users_alone(self): # STATE decides what is re-armed self.assertEqual(self.update(self.fake(after=None))[0], Q.SILENT) self.assertEqual(os.stat(self.q.state).st_mode & 0o777, 0o600) + def test_a_leftover_temp_file_does_not_give_the_plan_its_mode(self): # write_text's fchmod + os.makedirs(os.path.dirname(self.q.state), mode=0o700, exist_ok=True) + with open(self.q.state + '.tmp', 'w') as f: + f.write('half a plan') + os.chmod(self.q.state + '.tmp', 0o666) + Q.write_text(self.q.state, json.dumps(self.plan())) + self.assertEqual(os.stat(self.q.state).st_mode & 0o777, 0o600) + + def test_a_link_at_the_temp_file_is_never_written_through(self): # write_text's O_NOFOLLOW + os.makedirs(os.path.dirname(self.q.state), mode=0o700, exist_ok=True) + target = os.path.join(self.w.dir, 'elsewhere') + with open(target, 'w') as f: + f.write('keep') + os.symlink(target, self.q.state + '.tmp') + with self.assertRaises(OSError): + Q.write_text(self.q.state, json.dumps(self.plan())) + with open(target) as f: + self.assertEqual(f.read(), 'keep') + self.assertFalse(self.plan_exists()) + + def test_a_link_at_the_plan_is_kept_aside_and_not_followed(self): # recover's O_NOFOLLOW + target = os.path.join(self.w.dir, 'a-plan-elsewhere') + Q.write_text(target, json.dumps(self.plan())) # this user's, 0600: followed, it would be trusted + os.makedirs(os.path.dirname(self.q.state), mode=0o700, exist_ok=True) + os.symlink(target, self.q.state) + f = self.fake(armed=False, h1=self.up) + self.q.gh = f + code, out = run_quiet(self.q.recover) + self.assertEqual((code, f.calls, f.armed), (1, [], None), out) + self.assertIn('ERROR: unreadable', out) + self.assertTrue(os.path.islink(self.q.state + '.bad')) + with open(target) as f: + self.assertEqual(json.load(f), self.plan()) + def test_a_plan_another_user_could_write_is_kept_aside_and_not_followed(self): f = self.fake(armed=False, h1=self.up) Q.write_text(self.q.state, json.dumps(self.plan())) @@ -924,7 +966,8 @@ def test_a_planted_hook_never_runs(self): self.assertFalse(os.path.exists(marker), 'a hook in the cache ran with the token in its environment') def test_a_cache_with_grafts_shallow_or_alternates_is_rebuilt(self): # each changes what the objects say - for name in ('info/grafts', 'shallow', 'objects/info/alternates', 'objects/info/http-alternates'): + for name in ('info/grafts', 'shallow', 'objects/info/alternates', 'objects/info/http-alternates', + 'info/attributes'): # attributes: `* merge=union` makes a conflict merge clean self.assertTrue(self.fetch(self.c)[0]) path = os.path.join(self.q.cache, name) os.makedirs(os.path.dirname(path), exist_ok=True) @@ -942,11 +985,13 @@ def test_a_cache_with_another_config_or_open_to_others_is_rebuilt(self): got, out = self.fetch(self.up) self.assertTrue(got, out) self.assertIn('has a config the queue did not write: rebuilding it', out) - os.chmod(self.q.cache, 0o770) - got, out = self.fetch(self.c) - self.assertTrue(got, out) - self.assertIn("is not this user's alone", out) - self.assertEqual(os.stat(self.q.cache).st_mode & 0o777, 0o700) + for mode in (0o770, 0o755): # group-writable, or only readable by others: both are rebuilt + with self.subTest(mode=oct(mode)): + os.chmod(self.q.cache, mode) + got, out = self.fetch(self.c) + self.assertTrue(got, out) + self.assertIn("is not this user's alone", out) + self.assertEqual(os.stat(self.q.cache).st_mode & 0o777, 0o700) def own(comment): @@ -989,6 +1034,20 @@ def test_a_red_that_only_a_cancelled_run_followed_still_stands(self): def test_a_green_that_a_cancelled_run_followed_is_not_red(self): self.assertIsNone(self.failing(('2026-11-09T01:00:00Z', 'SUCCESS'), ('2026-11-09T02:00:00Z', 'CANCELLED'))) + def test_a_red_a_green_re_run_then_a_cancelled_run_is_not_red(self): # r3 corr minor (:283) + self.assertIsNone(self.failing(('2026-11-09T01:00:00Z', 'FAILURE'), ('2026-11-09T02:00:00Z', 'SUCCESS'), + ('2026-11-09T03:00:00Z', 'CANCELLED'))) + self.assertEqual(self.failing(('2026-11-09T01:00:00Z', 'SUCCESS'), ('2026-11-09T02:00:00Z', 'FAILURE'), + ('2026-11-09T03:00:00Z', 'CANCELLED')), 'review-check', 'red after the green') + + def test_the_newest_run_of_a_name_counts_whichever_workflow_ran_it(self): # r3 corr minor (:279), required_now + old_red = dict(self.rollup(('2026-11-09T01:00:00Z', 'FAILURE'))[0], workflowName='a') + new_green = dict(self.rollup(('2026-11-09T02:00:00Z', 'SUCCESS'))[0], workflowName='b') + self.assertIsNone(self.q.failing({'statusCheckRollup': [old_red, new_green]})) + self.assertEqual(self.q.required_now([old_red, new_green]), {'review-check': 'ok'}) + new_red = dict(old_red, startedAt='2026-11-09T03:00:00Z') + self.assertEqual(self.q.failing({'statusCheckRollup': [new_red, new_green]}), 'review-check') + def test_a_check_that_is_not_required_is_ignored(self): self.assertIsNone(self.failing(('2026-11-09T02:00:00Z', 'FAILURE'), name='lint')) @@ -998,11 +1057,12 @@ def test_two_red_checks_name_the_first_by_name(self): self.assertEqual(self.q.failing({'statusCheckRollup': rollup}), 'build') def sweep(self, state='BEHIND', runs=(('2026-11-09T02:00:00Z', 'FAILURE'),), comments=(), head=None, dry=False, - fail=(), name='review-check'): + fail=(), name='review-check', pushed=None): + """head: the PR's head in the list; pushed: its head when tell_red reads it (a push in between).""" self.posted, self.updated = [], [] listed = [{'number': 7, 'title': 't', 'isDraft': False, 'autoMergeRequest': {'m': 1}, 'mergeStateStatus': state, 'createdAt': '2026-11-07T00:00:00Z', 'statusCheckRollup': self.rollup(*runs, name=name), - 'baseRefName': 'main'}] + 'headRefOid': head or self.H, 'baseRefName': 'main'}] def gh(*a): if a[:2] == ('pr', 'list'): @@ -1010,7 +1070,7 @@ def gh(*a): if a[:2] == ('pr', 'view'): if 'view' in fail: return failed('boom') - return ok(json.dumps({'headRefOid': head or self.H, 'comments': [own(c) for c in comments]})) + return ok(json.dumps({'headRefOid': pushed or head or self.H, 'comments': [own(c) for c in comments]})) if a[:2] == ('pr', 'comment'): self.posted.append(a[-1]) return failed('boom') if 'comment' in fail else ok() @@ -1057,6 +1117,13 @@ def test_it_is_told_once_per_head(self): self.sweep(comments=[first], head='e5' * 20) self.assertEqual(len(self.posted), 1, 'a new head that is red is a new message') + def test_a_push_after_the_list_posts_nothing(self): # r3 corr minor (:356): the red is not known on the new head + new = 'e5' * 20 + code = self.sweep(pushed=new) + self.assertEqual((code, self.posted), (0, [])) + self.sweep(head=new) + self.assertIn(f'is red on {new[:12]}', self.posted[0], 'the next run, listing the new head, tells it') + def test_only_the_queues_own_comment_counts_as_told(self): # anyone can post a copy of the mark self.sweep() first = self.posted[0] @@ -1081,7 +1148,7 @@ def test_an_older_red_no_longer_skips_the_pr(self): def test_the_message_can_never_count_as_a_lens(self): self.sweep() - post = {'body': self.posted[0], 'author': {'login': 'owner'}} + post = note(self.posted[0], 't') # an author whose lens comments count: a look-alike would block (NO_HEADER) self.assertIsNone(REVIEW.lens_problem(post)) self.assertEqual(REVIEW.reviewed_head({'comments': [post], 'reviews': []}), (False, None, 'no lens comments')) @@ -1166,11 +1233,32 @@ def test_a_re_run_of_the_cancelled_check_holds_the_queue_again(self): # the ne self.assertIn('#1 is up to date and waiting', out) def test_green_or_running_checks_still_hold_the_queue(self): - for first in (self.checks(), self.checks(review_check='IN_PROGRESS'), []): + for first in (self.checks(), self.checks(review_check='IN_PROGRESS')): with self.subTest(first=len(first)): code, out = self.sweep(first) self.assertEqual((code, self.updated, self.posted, self.asked), (0, [], [], []), out) + def test_a_cancelled_check_waits_while_a_workflow_runs_on_the_head(self): # r3 corr minor (:322) + # cancel-in-progress: the newer run of the workflow is queued and has no job yet, so it is not in the rollup + for kw in ({'runs': '[{"status": "queued", "workflowName": "ci"}]'}, {'runs_fail': True}): + with self.subTest(**kw): + code, out = self.sweep(self.checks(build='CANCELLED'), **kw) + self.assertEqual((code, self.updated, self.posted, self.asked), (0, [], [], [self.H1]), out) + self.assertIn('required check build ended CANCELLED, and which workflows still run', out) + + def test_a_head_no_check_ever_reported_on_is_told_once_its_workflows_ended(self): # r3 corr minor (:328) + # every job skipped itself or the workflow ended before any required job: it never merges, so it must not hold + code, out = self.sweep([], runs='[{"status": "completed", "workflowName": "ci"}]') + self.assertEqual((self.asked, self.updated, len(self.posted)), ([self.H1], [2], 1), out) + self.assertIn('required check build never reported', self.posted[0]) + for kw in ({'runs': '[]'}, {'runs': '[{"status": "in_progress", "workflowName": "ci"}]'}, {'runs_fail': True}): + with self.subTest(**kw): # nothing ran yet, or still runs, or unknown: it may still get its checks + code, out = self.sweep([], **kw) + self.assertEqual((code, self.updated, self.posted), (0, [], []), out) + self.assertIn('is unknown: waiting', out) + code, out = self.sweep([], runs='[]') + self.assertIn('no check and no workflow run on its head yet: waiting', out) + def test_a_required_check_that_never_reports_is_told_once_nothing_runs(self): code, out = self.sweep(self.checks(test=None), runs='[{"status": "completed", "workflowName": "ci"}]') self.assertEqual((self.asked, self.updated, len(self.posted)), ([self.H1], [2], 1), out) @@ -1199,6 +1287,43 @@ def test_a_dry_run_posts_nothing_and_says_what_it_skips(self): self.assertIn('skip #1: up to date, but required check build ended CANCELLED', out) +class UnknownMergeState(unittest.TestCase): + """GitHub shows UNKNOWN for a few seconds after a push to the base, when the push's queue run starts: the older PR + is listed again until its state is known, so a younger one is not updated ahead of it (r3 corr minor, :518).""" + def setUp(self): + d = tempfile.mkdtemp(prefix='pr-queue-unknown-') + self.addCleanup(shutil.rmtree, d, True) + self.sleeps = [] + self.q = make_queue(d, sleeps=self.sleeps) + + def sweep(self, unknown_lists): + self.updated, lists = [], [] + + def gh(*a): + if a[:2] == ('pr', 'list'): + lists.append(1) + first = 'UNKNOWN' if len(lists) <= unknown_lists else 'BEHIND' + return ok(json.dumps([{'number': n, 'title': 't', 'isDraft': False, 'autoMergeRequest': {'m': 1}, + 'mergeStateStatus': first if n == 1 else 'BEHIND', 'statusCheckRollup': [], + 'createdAt': f'2026-11-07T00:00:0{n}Z', 'headRefOid': f'{n}' * 40, + 'baseRefName': 'main'} for n in (1, 2)])) + raise AssertionError(a) + self.q.gh = gh + self.q.update = lambda n, d: self.updated.append(n) or 0 + return run_quiet(self.q.run, False) + + def test_an_older_pr_in_unknown_state_is_listed_again_before_a_younger_one_moves(self): + code, out = self.sweep(unknown_lists=2) + self.assertEqual((code, self.updated, self.sleeps), (0, [1], [CFG.poll_seconds] * 2), out) + self.assertIn('#1: merge state UNKNOWN', out) + + def test_an_unknown_state_that_stays_is_passed_over_after_wait_seconds(self): + tries = CFG.wait_seconds // CFG.poll_seconds + code, out = self.sweep(unknown_lists=tries + 5) + self.assertEqual((code, self.updated, len(self.sleeps)), (0, [2], tries), out) + self.assertIn('skip #1: merge state UNKNOWN', out) + + class RedChecksWithTheBaseRule(RedChecks): """Every RedChecks case again with update_when_base_was_red on: a red whose run names no tested base (its job has no tested-base step) is the PR's own, as with the rule off.""" @@ -1349,6 +1474,27 @@ def test_a_red_of_its_own_beside_a_base_still_red_is_its_authors(self): # r3 m self.assertNotIn('not this PR', self.posted[0]) self.assertIn('Fix the cause and push', self.posted[0]) + def test_the_notice_names_the_prs_own_red_beside_a_base_red_that_sorts_first(self): # r4 corr minor (:737) + main = {k: {**v, 'test': 'SUCCESS'} for k, v in self.main_of(tip='FAILURE').items()} # build: the base's + code, out = self.one({'build': [(self.RUN, 'FAILURE', self.M1)], 'test': [(self.RUN, 'FAILURE', self.M1)]}, main) + self.assertEqual((self.updated, len(self.posted)), ([], 1), out) + self.assertIn(f'required check test is red on {self.H[:12]}', self.posted[0]) + self.assertIn(f'skip #7: required check test failed and an update would not fix it yet', out) + + def test_the_wait_notice_is_told_again_when_the_check_it_names_turns_green_on_the_base(self): # r4 (:729/:361) + runs = {'build': [(self.RUN, 'FAILURE', self.M1)], 'test': [(self.RUN, 'FAILURE', self.M1)]} + both = self.main_of(tip='FAILURE', other={'test': 'FAILURE'}) # build and test red on main + self.one(runs, both) + first = self.posted[0] + self.assertIn('once build is green on main', first) + self.one(runs, both, comments=[first]) + self.assertEqual(self.posted, [], 'told once per head and check') + fixed = self.main_of(other={'test': 'FAILURE'}) # main fixed build; test is still red there + code, out = self.one(runs, fixed, comments=[first]) + self.assertEqual((self.updated, len(self.posted)), ([], 1), out) + self.assertIn(f'required check test failed as on main on {self.H[:12]}', self.posted[0]) + self.assertIn('once test is green on main', self.posted[0]) + def test_the_wait_notice_names_the_check_that_is_red_on_the_base(self): # r3 minor: not just the first red main = self.main_of(other={'test': 'FAILURE'}) # build: red on M1, green now; test: red on M1 and now code, out = self.one({'build': [(self.RUN, 'FAILURE', self.M1)], 'test': [(self.RUN, 'FAILURE', self.M1)]}, main)