Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 53 additions & 26 deletions fork-ci/actions/pr-queue/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,6 @@ up to date with their base branch. After each push to the base branch it brings
exactly one waiting pull request up to date, and it turns auto-merge back on
only for a head that holds exactly the change the review read.

This folder is built in four stacked pull requests:

1. the review rules and the config (`lens.py`, `config.py`);
2. the queue itself (`pr_queue.py`, section "The queue");
3. armed pull requests that cannot merge (a red required check, or one that
ended without passing or never reported) are told once and passed over;
4. the "red only because the base was red" rule, the reusable workflow
`.github/workflows/fork-pr-queue.yml`, `action.yml`, `tested-base/` and the
usage sections of this file: this part.

## Config

One JSON file in the project's repository. Unknown keys, a key written twice, a
Expand Down Expand Up @@ -93,6 +83,12 @@ heading near the top, or two `covered:`/`checks:` lines. A comment that is no
report says so on its first line, `not a lens: ...`. That clears it unless one of
its lines is a lens header or a `lens:` field with a sha.

A comment that looks like a report but is not one valid read is a lens of its
own, keyed by that comment. Posting the report again the right way adds a new
read and leaves the old comment blocking, and hiding it is not enough either.
The ways out: delete it, edit its first line to `not a lens: ...` with no lens
header line left in it, or post an override for the head.

## The queue

```
Expand All @@ -109,26 +105,38 @@ first, and acts on the first one it can:
the way out.

A pull request in a merge state the queue neither updates nor waits for (a
conflict, `DIRTY`) is passed over, and the run says so.
conflict, `DIRTY`) is passed over, and the run says so. `UNKNOWN` (GitHub is
still computing it, often for a few seconds after a push to the base) is listed
again every `poll_seconds`, for at most `wait_seconds`, before it is passed
over, so a younger pull request is not updated ahead of it.

An armed pull request that cannot merge does not hold the queue. It is told
why on the pull request, once per head, check and reason, and passed over:

- a required check whose newest run failed (`FAILURE`, `ERROR`, `TIMED_OUT`,
or `CANCELLED` after a red with nothing green since), behind or not. Its
author fixes it; nothing in the queue re-runs it.
or `CANCELLED` after a red with nothing green since), behind or not. The
newest run of the name counts, whichever workflow ran it, as the ruleset
reads it. Its author fixes it; nothing in the queue re-runs it.
- up to date, and a required check's newest run ended without passing and
without a red (`CANCELLED`, `STARTUP_FAILURE`, `STALE`, ...): nothing
without a red (`CANCELLED`, `STARTUP_FAILURE`, `STALE`, ...), and no
workflow runs on the head any more (a newer run may still be queued): nothing
re-runs it by itself. The comment says to re-run the CI run or push.
- up to date, and a required check never reported although no workflow runs
on the head any more. The comment says to push or merge the base: a re-run
repeats the jobs of the workflow files the head was built with, so it cannot
add a check those files do not have (one added or renamed since).
on the head any more, also when no check at all reported. The comment says
to push or merge the base: a re-run repeats the jobs of the workflow files
the head was built with, so it cannot add a check those files do not have
(one added or renamed since).

A head with no check and no workflow run at all still holds the queue: nothing
tells "not started yet" from "never starts" (every workflow filtered out by
its paths) without a clock, and the queue keeps no state for one.

Only the newest run of a check counts, and a check not in `required_checks` is
ignored. Only the queue's own comment (posted with its token) counts as told:
a copy of it by anyone else does not silence it. When GitHub cannot say which workflows still run on the head, the
pull request is not called stuck: it holds the queue, and the run says why.
a copy of it by anyone else does not silence it. A pull request whose head
moved between the list and the comment is told by the next run, on its new
head. When GitHub cannot say which workflows still run on the head, the pull
request is not called stuck: it holds the queue, and the run says why.

`gh` must be logged in for the repository (`GH_TOKEN`). Git fetches into the
queue's own bare repository (`--cache`) and gets its login from
Expand All @@ -140,8 +148,9 @@ other repository's job, as a user of its own. The `--state-file` and `--cache`
folders belong to that user alone. git runs with no hooks, no fsmonitor and no
replace refs, and a cache that is not that user's alone (mode `0700`), has a
config the queue did not write, or holds a file that changes what its objects
say (`info/grafts`, `shallow`, `objects/info/alternates` or
`objects/info/http-alternates`) is rebuilt, never used. A job of the same user
say or how a merge reads them (`info/grafts`, `shallow`,
`objects/info/alternates`, `objects/info/http-alternates` or
`info/attributes`) is rebuilt, never used. A job of the same user
can still change the cache between that check and git's next read: only the
runner rule above stops that.

Expand Down Expand Up @@ -169,7 +178,15 @@ was: give each queue its own `--state-file`. A plan without `repo` (the source
queue's format) is this repository's. An unreadable plan, or one another user
could have written (not this user's, or writable by its group or others), is
kept as `STATE.bad`, the run fails, and every later run warns while that file
exists.
exists. A plan does not say who turned auto-merge off: a pull request a person
disarmed after the stop is re-armed too, if its head holds the change read.

The plan is on disk, and that is why the runner rule above exists. It could
live in a comment of the queue's own login instead (trusted by
`viewerDidAuthor`, as the told notices are): the next run reads the review and
the base from GitHub anyway, and a failed write before the disarm would only
mean no disarm. That stays open for a queue that moves several pull requests at
once, and for GitHub-hosted runners.

`--lock-file` (default: `STATE.lock`) keeps two runs from overlapping.
`--dry-run` takes no lock and writes nothing: no `STATE`, no cache, no comment.
Expand All @@ -191,7 +208,10 @@ red required checks is therefore updated when, for every red required check:

When the base tip is not green yet, the pull request is told once, per head
and check, that it failed as on its base and that the queue updates it by
itself once the base is green. It does not hold the queue meanwhile. When
itself once the base is green. The notice names the first check still red on
the base; when the base fixes that one and another is still red there, a new
notice names that one. A red of the pull request's own beside a base red is
named as its own ("fix the cause"). It does not hold the queue meanwhile. When
anything is unknown (no note among the job's first 100 annotations, a failed read,
more than 100 checks on a commit), the red counts as the pull request's own, as without
the rule. There is no state and no loop: after the update the new runs test a
Expand Down Expand Up @@ -300,12 +320,14 @@ replaced, and the replaced corpus was checked against the source copy before it
was committed. A rule change here must keep the corpus green or say in its pull
request why a recorded answer changes.

The port differs on purpose in three ways, and the replay checks each recorded
The port differs on purpose in four ways, and the replay checks each recorded
answer that changes for its reason. The source's repository is private, so its
comments carry no author association: the replay gives them `OWNER` and
unedited. A read shorter than 40 hex, and a lens comment edited after posting,
read nothing here (54 recorded answers). The source's `full_sha`, which matched
a short read to the pull request's commits, is gone.
a short read to the pull request's commits, is gone. A hidden (minimized) lens
comment reads nothing: the source never read `isMinimized`, and no recorded
call has it, so no recorded answer changes for it.

`pr_queue.py` is a port of the same project's queue, and `pr_queue.test.py`
holds that project's queue tests with the port's names. What the port changed:
Expand All @@ -317,7 +339,12 @@ queue's own advice in a comment keeps its backticks and `<sha>` placeholder
required check that never reported, the comment says to push or merge the
base; the source said to re-run, which cannot add a check the head's workflow
files do not have. With two red checks the comment names the first by name
(the source named whichever the check list gave first).
(the source named whichever the check list gave first). A red check is the
newest run of its name whichever workflow ran it, and a red that a green re-run
and then a cancelled run followed is not red (the source took the newest run
per workflow, and counted any red before a cancel). An `UNKNOWN` merge state is
listed again (the source passed it over at once), and a cancelled check waits
while a workflow still runs on the head (the source called it stuck at once).

The "base was red" rule is a port of a change proposed to the source queue,
with its tests. What the port changed: the rule is off unless the config turns
Expand Down
47 changes: 34 additions & 13 deletions fork-ci/actions/pr-queue/lens.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,17 @@
# The name ends at the first field: ` · reviewer:` or ` · read at:` (also `read at <hex>`; the separator may also be
# `,` `—` or none). A bare `reviewer` or `read at` stays in the name (`data-integrity reviewer 1` and `... 2` are two
# lenses).
FIELD_RE = re.compile(r'\s*(?:[·,—]\s*)?\b(?:reviewer\s*:|read[\s-]*at\b\s*(?::|[0-9a-f]{7}))', re.I)
READ_AT_RE = re.compile(r'\bread[\s-]*at\b\s*:?\s*([0-9a-f]{7,40})\b', re.I)
# Every pattern here runs in time linear in the line, also for a trusted author's 64 KB comment: a blank run is tried
# once, from its start ((?<!\s)), and `\s*(?::\s*)?` never splits one blank run between two `\s*`.
FIELD_RE = re.compile(r'(?<!\s)\s*(?:[·,—]\s*)?\b(?:reviewer\s*:|read[\s-]*at\b\s*(?::|[0-9a-f]{7}))', re.I)
READ_AT_RE = re.compile(r'\bread[\s-]*at\b\s*(?::\s*)?([0-9a-f]{7,40})\b', re.I)
FULL_SHA = re.compile(r'[0-9a-f]{40}')
NO_HEADER = 'comment without a first-line lens header'
# The ways out of a NO_HEADER entry (lens_entries), for the queue's advice. A correct re-post adds a new entry and
# leaves the old one; hiding the old one is not enough either.
# The queue posts this text, so it never holds the lens marker with its colon (no_lens_text).
NO_HEADER_WAY_OUT = ('delete that comment (hiding it is not enough), or edit it to start with `not a lens` and a colon '
'and hold no lens header line, or')
EDITED = 'edited after posting: post a new comment instead'
HIDDEN = 'hidden: post a new comment instead'
SHORT = 'not a full 40-hex sha'
Expand Down Expand Up @@ -84,12 +91,25 @@ def lens_key(name):
marker (`r2`, `re-check`, `lens`) and Markdown emphasis around it. Everything else is the lens's identity:
'data-integrity (DI-1)' and 'data-integrity (DI-2)' are different lenses; 'data-integrity DI-1 r2' =
'Data-integrity DI-1'."""
n, prev = name.lower(), None
while n != prev:
prev = n
n = ' '.join(n.split()).strip('*_`')
n = re.sub(r'\s+(r\d+|lens|re-?check)$', '', n)
return n or 'unnamed'
# The old loop (join the words, strip `*_\`` from both ends, drop one trailing ` <marker>`, until nothing changes)
# copied the whole name per pass: minutes for a 64 KB name of 13 K markers. The same steps on a word list, where a
# word stripped empty at an end stays as the blank in front of a marker until the next pass, as in the text.
words = name.lower().split()
lo, hi = 0, len(words)
while True:
while lo < hi and not words[lo]:
lo += 1
while lo < hi and not words[hi - 1]:
hi -= 1
if lo == hi:
return 'unnamed'
before = (words[lo], words[hi - 1], hi)
words[lo] = words[lo].lstrip('*_`')
words[hi - 1] = words[hi - 1].rstrip('*_`')
if hi - lo >= 2 and words[hi - 1] and re.fullmatch(r'r\d+|lens|re-?check', words[hi - 1]):
hi -= 1
if (words[lo], words[hi - 1], hi) == before:
return ' '.join(w for w in words[lo:hi] if w) or 'unnamed'


def names_a_lens(name):
Expand Down Expand Up @@ -266,11 +286,12 @@ def lens_posts(self, pr):
def lens_entries(self, pr):
"""{lens key: ((when, index), sha or None, why or None)}: each lens's NEWEST header, from the posts that count
(counts). A comment that looks like a lens report but is not one valid read (lens_problem) is an entry of its
own with no sha: it read nothing until it is re-posted as one comment per lens with a first-line header, or an
override covers it. A lens comment edited after posting (includesCreatedEdit not false: GitHub records that it
changed, not what) or hidden (minimized) read nothing until that lens posts a new comment. A lens-looking post
that does not count is logged (log only, never a comment), so a wrong lens_author_associations shows in the
run."""
own, keyed by the comment, with no sha: a correct new post of that lens does not replace it, and hiding it
does not clear it. It reads nothing until it is deleted, its first line is edited to `not a lens: ...` with no
lens header line left in it, or a newer override covers it (NO_HEADER_WAY_OUT). A lens comment edited after
posting (includesCreatedEdit not false: GitHub records that it changed, not what) or hidden (minimized) reads
nothing until that lens posts a new comment. A lens-looking post that does not count is logged (log only, never
a comment), so a wrong lens_author_associations shows in the run."""
found = {}
for when, i, post in self.lens_posts(pr):
where = post.get('url') or f'by {(post.get("author") or {}).get("login") or "?"} at {when or "?"}'
Expand Down
Loading
Loading