[MaxGhenis]

Fixes #712

What and why

Subfleet build jobs keep their state in a .job/ folder at the root of their worktree: HANDOFF.md, PR_BODY.md, env, a HEAD file, git bundles and sometimes a whole repo.git. On 23 October 2026, #704 committed .job/HANDOFF.md, and #699 committed .job/HANDOFF.md and .job/PR_BODY.md. Later commits on both branches removed them. Nothing in CI would have stopped them. pr_code_changes.yaml has a paths: filter, so a pull request that adds only such files runs no workflow at all.

This PR adds an Agent notes workflow (.github/workflows/pr_agent_notes.yaml). It runs .github/check-agent-notes.sh on every pull request and fails when the PR's head commit contains:

• a .job/, .job-git/ or .diag/ folder at any depth, or a file of one of those names. .job-git/ is a Subfleet job's git directory. A local clone of policyengine-uk has 674 .job-git/ files and 8 .diag/ files in its branch history, none ever on main.
• a root-level HANDOFF.md, handoffs.jsonl, PR_BODY.md, PR-BODY.md, REPORT.md or PROGRESS.md, in any ASCII capitalisation, so pr-body.md and pr_body.md count too. Four of these (REPORT.md, PROGRESS.md, HANDOFF.md, handoffs.jsonl) are policyengine-uk's list.

It also ignores the three folders in .gitignore, so git add -A leaves them out, and documents the check in docs/engineering/skills/github-prs.md.

Design

policyengine-uk's "No agent working notes at the repository root" step (pr_code_changes.yaml on its main) is the model, and its pieces carry over:

• Pathspecs. git ls-files with :(top,icase) pathspecs, plus :(glob,icase)**/ for the folders at any depth.
• Failure handling. shell: bash adds pipefail, and the script also sets set -euo pipefail, so a git failure fails the step instead of passing it. The script reports a failure with ::error:: and exit 1.
• Cancellation. A new push or edit to a pull request cancels its run in progress; the newer run checks the current head in full. Nothing cancels a run on main.

It departs from policyengine-uk's guard where pe.py differs, or where review found a gap:

• Its own workflow, with no path or branch filter, and edited handled. policyengine-uk's guard is a step in Lint, which runs on every pull request there. pe.py's code workflow has a path filter, so the guard needs its own workflow. types: [opened, synchronize, reopened, edited]: retargeting emits edited, so the guard checks a stacked PR whose old base had no guard when someone retargets it to main. Every edit runs the full check on the head, with no condition, so a title edit cannot replace a failure with a skipped run. As with every pull_request workflow, GitHub runs it only from the workflow files of the PR's merge with its base, and not at all while the PR conflicts with its base.
• It runs the default branch's guard. On a pull request, the job fetches the default branch along with the head. It copies the default branch's .github/check-agent-notes.sh into $RUNNER_TEMP and runs that copy, so a PR cannot pass by editing the script. Round 3 reproduced a false pass with a pass-always script. Only while the default branch has no guard, on this PR, does the checked-out copy run. A missing default-branch ref fails the step rather than falling back. Push, schedule and dispatch runs check out the commit they check. The guard cannot stop a PR that edits the workflow file itself without pull_request_target or a required-workflow setting; the edit shows in the PR's diff.
• It checks the PR's head commit, not the merge with the base. The job fetches github.event.pull_request.head.sha by exact SHA. The script reads <commit>^{tree} into a temporary GIT_INDEX_FILE and lists it with the same pathspecs. The result does not depend on the base, so retargeting a stacked pull request cannot change it. No commit on main's first-parent line, all 510 back to the initial commit, has had one of these paths in its tree. The merges of Add canonical SPM selection, receipts and native loading to the wrapper #515, Allow a UK local-area release on an accepted model mismatch #699 and Support bundle-pinned local data release manifests #704 bring 61 commits into main's history that did (44 with a root PROGRESS.md, 17 with .job/ notes), but none reached main's tree. So a PR fails exactly when its head holds a note, and for a PR cut from main's first-parent line, that means exactly when its diff adds one. A branch whose merge base with main is one of the 61 commits, as Build and verify isolated wrapper release candidates #520's is, can carry a note that its diff shows as modified rather than added; the check fails it too.
• It also checks main on every push and once a day. A skip instruction in a pushed commit's message skips pull_request and push runs. GitHub's docs: "Skip instructions only apply to the push and pull_request events." A merge that carries such a commit to main can skip the push run too. This PR showed it by accident: two of its own head commits quoted the instruction while describing this gap, and no pull request workflow ran on them. No commit message can skip the daily schedule run, so the next scheduled run finds a note that reaches main that way, if it is still there. GitHub can delay or drop a scheduled run. The push run catches an ordinary direct push at once. GitHub disables schedules in public repositories after 60 days without activity; this one is active.
• pull_request, not pull_request_target. Round 2 used pull_request_target. That made skip instructions powerless, ran on conflicted PRs, and ran main's copy of the guard. Two things ruled it out. GitHub blocks pull_request_target in public repositories without an Actions event policy from 2 December 2026, and neither this repository nor the org has one. And pe.py's PR verification never uses it (docs/release-bundles.md). Allowing it is a settings decision, outside this PR.
• Output that can't forge workflow commands. A path name comes from the pull request. The runner reads ::name:: at the start of a line, and the legacy ##[name] anywhere in a line. So the guard prints the paths between ::stop-commands::<random 32-hex token> and ::<token>:: (the runner masks the token in the log), and it percent-encodes % in the annotation data. The log lists every path; the annotation gives the count and the first path. This handles a tracked repo.git.
• A script, not inline, like .github/check-changelog.sh, so pytest can drive it. It runs under the macOS bash 3.2 as well as Ubuntu's bash 5.
• permissions: contents: read, no secrets, ubuntu-latest. Let CI test and check jobs use another runner pool, set by repository variables #706's runner-variable test (invariant 2: every other job keeps ubuntu-latest) still holds.
• UK certification. scripts/check_uk_candidate.py promotes a candidate automatically only once the newest run of every workflow on its pull request has succeeded, so automatic promotion now also waits for this check. The check runs when the certification workflow opens a candidate PR, pushes a new candidate commit to one, or edits its body, all with its GitHub App token, and takes seconds. A promote run that starts before it finishes waits for the next scheduled run, as it does for any unfinished check.

Invariants

1. The guard fails exactly when the tree holds a note, as defined above. It reports every such path and no other.
2. Given a commit, it judges that commit's tree, whatever the index holds, and leaves the index as it was.
3. A git failure fails the guard instead of passing it.
4. The runner reads no path the guard prints as a workflow command. The only commands it emits are its own stop-commands, the matching resume and one error.
5. The workflow has no path or branch filter, and it runs on every open, push, reopen and edit (a retarget included) of a pull request. So it checks the head of every pull request that GitHub runs workflows for: one whose merge with its base holds this workflow and has no conflict. It also checks main on every push and on a schedule, which no commit message can skip. It has a read-only token and no pull_request_target.
6. On a pull request, it runs the default branch's guard, so editing the script cannot pass the PR. Only while the default branch has no guard does the checked-out copy run.

Test plan

• tests/test_check_agent_notes.py, 55 tests:

  • the sample .job/HANDOFF.md fails, and a normal change passes;
  • 21 notes fail, and 13 near misses pass (docs/HANDOFF.md, .jobs/, x.job/, my.job, REPORTS.md and others);
  • the guard reports a tracked repo.git in full;
  • commit mode judges the commit's tree, not the index, and leaves the index unchanged;
  • command-shaped names (##[stop-commands]…, ##[add-mask]…, ::warning::…, an indented ::error::…) and newline names, which an emulation of the runner's parser finds no command in. The emulation follows actions/runner source (ActionCommand.cs TryParseV2/TryParse, ActionCommandManager.cs TryProcessCommand);
  • a % in the annotation;
  • git failures: outside a repository, and an unknown commit;
  • this repository's index and HEAD, and the .gitignore rules;
  • the workflow's triggers, steps and token;
  • the workflow's own guard step. The test reads it from the YAML and runs it as GitHub runs a bash step, in scratch repositories that hold what the job's earlier steps leave. A PR with a pass-always script and .job/HANDOFF.md still fails. Before main has a guard, the checked-out copy runs. A missing default-branch ref fails closed. Push, schedule and dispatch events check their own commit;
  • a Hypothesis differential test on 150 generated trees. It checks the index mode against a Python reference, and the commit mode against the index mode. The trees mix glob characters, colons, non-ASCII letters and random case.

• Mutation check. Fourteen mutants each fail the suite:

  • case folding dropped for .diag;
  • .job matched at the root only;
  • PR-BODY.md dropped;
  • set -e and pipefail removed;
  • the line prefix removed;
  • note names matched at any depth;
  • stop-commands removed;
  • the commit read into the real index;
  • the commit argument ignored;
  • the % encoding removed;
  • main's copy never used;
  • the fail-closed default-branch check removed;
  • edited dropped;
  • the PR's copy preferred over main's.

• A simulation ran the guard's steps against the head of every open PR, on 23 October and again on 5 November: a clone of main, then a fetch of each open PR's head and the script on that SHA. Both times only Add behavior input binding contract #511 and Build and verify isolated wrapper release candidates #520 fail (both PROGRESS.md). On 5 November the other 29 open PRs, this one among them, pass.

• Against main as of 5 November (286fce481, 54 commits past this PR's base), on a local merge of this head:

  • the guard passes on the merged tree;
  • tests/test_check_agent_notes.py and Let CI test and check jobs use another runner pool, set by repository variables #706's tests/test_ci_runner_variables.py pass, 130 tests;
  • shellcheck on the script, and ruff check and ruff format --check on the test file, are clean;
  • the guard passes on all 510 commits of main's first-parent line, from a full-history clone.

• actionlint 1.7.12 on pr_agent_notes.yaml: no findings. On all workflows, its one finding is the existing actions/configure-pages@v3 in push.yaml, which this PR does not touch.

• This version on GitHub:

  • it passes on 693d012 (run 40015348571);
  • it fails on a demonstration commit that adds .job/HANDOFF.md (run 40015409561), which the next commit reverts. Its log shows ::stop-commands::*** (the runner masks the token), then Agent or job note: .job/HANDOFF.md, then the annotation "1 agent or job note path(s) in the tree, starting with .job/HANDOFF.md". The .gitignore made the demonstration need git add -f.

  The round-1 version did the same (pass, fail). A squash merge keeps the demonstration commits off main.

• CI on the head 3f30759e5:

  • on 23 October, the latest run of every check passed (Code changes, Agent notes, Docs);
  • on 5 November, closing and reopening this PR reran CI on its merge with main at 286fce481 (merge commit 9d70dfcbb), and every check passed (Code changes, Agent notes, Docs).

• After merging, with a squash message free of skip instructions (the repository builds squash messages from commit messages, and 0fe7a2d's quotes one). The merge commit b672ca9de has the tree CI tested on 9d70dfcbb, and its message holds no skip instruction:

  • the workflow passes on main, on the push (run 41963910999) and on a dispatch (run 41963944172);
  • throwaway draft PR Throwaway: show the Agent notes check runs main's guard (do not merge) #740 added .job/HANDOFF.md and replaced the script with a pass-always one, and the check failed (run 41963984834). Its log shows main's copy reporting .job/HANDOFF.md, so main's copy ran. Throwaway: show the Agent notes check runs main's guard (do not merge) #740 is closed unmerged, and its branch is gone.

Review

GPT-6.1 Sol (subfleet run --task review --tier hard) reviewed rounds 1 to 5, and a separate Claude Opus 5.5 agent round 6.

• Round 1 (pull_request, merge tree): REQUEST_CHANGES.

  1. A base retarget did not re-run the check, so a stacked PR checked against a cleaned parent could pass.
  2. A skip instruction in the head commit skipped it.
  3. A path containing ##[stop-commands]token suppressed the error annotation.

  The fix: the check now reads the head tree, and output goes through stop-commands. The push and schedule runs on main narrow finding 2.

• Round 2 (pull_request_target): REQUEST_CHANGES. It found no bypass, and reproduced the retarget and stop-commands fixes, including with a compiled copy of the runner's parser. The one finding was the 2 December pull_request_target block. The fix returned to pull_request, as above.

• Round 3 (pull_request, head tree, push and schedule on main): REQUEST_CHANGES. It found no path-matching or output bypass, and verified the three GitHub runs. Its findings:

  1. A PR could pass by replacing the script; it reproduced a false pass.
  2. A retargeted old stacked PR got no first run, because the workflow did not handle edited.
  3. Nit: the docs said nothing can skip a push run, but a skip instruction can.

  8656ee4 fixes them: the job runs the default branch's guard and handles edited, and the docs say what a skip instruction can skip.

• Round 4 (8656ee4): APPROVE. No blocker or should-fix. It reproduced the closure of both round-3 findings in scratch repositories: main's copy runs over a PR's pass-always script, and the guard checks an old stacked head after a retarget to main. It verified the shallow two-refspec fetch, fork-object fetch and concurrency, and it caught three workflow-step mutations. One nit: "found within a day" was too strong, because GitHub can delay or drop scheduled runs. The next commit, 3f30759, fixes the wording and the concurrency comment it flagged.

• Round 5 (3f30759): APPROVE. The change since 8656ee4 touches only comments and one documentation paragraph, and the workflow YAML loads identically. The new wording agrees with GitHub's schedule and skip-instruction docs, actionlint passes, and it found no inaccurate mechanism claim in this description.

• Round 6 (Claude Opus 5.5; 3f30759 merged into main at 286fce4, and this description): REQUEST_CHANGES, on wording only. It reran the history check, the sweep, the tests on the merge, and the CI and documentation checks, and it found nothing wrong with the code or the merge. It found eight statements in this description and the squash message that were inaccurate or incomplete, and the effect on UK certification described above. This version corrects them all, and it approved the corrected text.

Sweep of open pull requests

A sweep on 5 November 2026, up to date at 03:50 UTC, matched the changed-file lists of all 31 open PRs (240 files, from gh api repos/PolicyEngine/policyengine.py/pulls/N/files) against every path this guard rejects. It also flagged any root-level .md, .jsonl or .txt file and any file name that mentions a hand-off, PR body, progress note, report or review. Then it ran the guard itself on every open PR's head commit, as in the test plan. The sweep on 23 October found the same two PRs.

PR	Branch	Finding	When the guard sees it
#511	feat/be-behavior-input-contract → main	adds root PROGRESS.md	fails on its next push, and on a reopen or edit once GitHub has refreshed its test merge with main; it merges cleanly into main, before and after this PR
#520	max/wrapper-release-candidates-20260914 → max/spm-canonical-wrapper-release-20260910	edits root PROGRESS.md, which its base has since deleted	no check runs yet. It conflicts with its base over PROGRESS.md, and with main over PROGRESS.md, push.yaml and two test files, so GitHub runs no pull_request workflow on it; neither its base nor its head has this workflow. Once its merge with its base holds the workflow and has no conflict, its next push or reopen runs the check, which fails if PROGRESS.md is still there

No open PR has a .job/, .job-git/ or .diag/ path, or a hand-off or PR-body file. The other root-level matches are edits to README.md (#581, #596, #709, #710). The sweep pushed nothing to any of these branches.

Not in this PR

• policyengine-uk's guard rejects root notes only. It does not reject .job/, .job-git/ or .diag/, and nothing checks its main on a schedule. That is a separate change to policyengine-uk.
• pe.py's main has no branch protection or ruleset, so nothing requires this check before a merge. Agents' merge gates (gh pr checks must pass) already block on it. A required check that a skip instruction skips stays pending and blocks the merge (GitHub's skip docs). Requiring it needs a ruleset that lets the release bot's version commits through, which is a settings decision.

axiom: n/a: CI infrastructure only

🤖 Generated with Claude Code