Repository navigation
Check that every PR description starts with Fixes #N for an open issue - #4177
Merged
Merged
Conversation
Port the simulation API's tests for check_pr_description.py and pr-description.yml (rule api.process.pr-tracking-issue), with four event payload fixtures: bundle-update PR #4169 without a Fixes line and fixed, PR #3776 with its Fixes line not first, and a description naming an issue in another repository. Test that the bundle-update script opens or reuses a tracking issue and puts Fixes #N first in the PR it opens. The check script and the workflow don't exist yet, and the bundle-update script writes no Fixes line, so all 59 cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the simulation API's PR description check (PolicyEngine/policyengine-sim-api#862). A new workflow, pr-description.yml, runs on every opened, edited, reopened and synchronize event of an open pull request, with no stack condition, and fails unless the description's first line is exactly Fixes #N and gh api, with the job's read-only token, reports N as an open issue in this repository. - Enforce api.process.pr-tracking-issue with the new job, with proof fixtures, and add pr-description.yml to the registry's discovered workflows. - The PolicyEngine bundle-update script now reuses the open "Update PolicyEngine bundle to <version>" issue or opens one, and starts the pull request's description with Fixes #N on both paths that open a pull request. The daily relock script already did. - Let CI/CD scripts under .github run other programs (S603, S607), as command-line scripts under scripts/ already may. - Describe the check in the PR skill, AGENTS.md and the README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4177 +/- ##
===========================================
+ Coverage 53.80% 88.30% +34.49%
===========================================
Files 281 293 +12
Lines 16310 17506 +1196
Branches 2639 2875 +236
===========================================
+ Hits 8776 15459 +6683
+ Misses 6803 1335 -5468
+ Partials 731 712 -19 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
27 of the 32 bundle-update PRs closed since 2026-10-01 were closed without merging, so an issue per version would be left open for each of them. Test that every bundle-update PR names the oldest open issue titled "Update the PolicyEngine bundle", or a new one, and that issues with other titles, including the per-version title, are not reused. - The fake gh now applies the script's --jq expression to a list of open issues with jq, so the tests exercise the selection and sort order. - run_update no longer passes the shell's GITHUB_REPOSITORY to the script, which made a test fail wherever that variable names another repository; a test passes one explicitly to check the script uses it. - Correct the comment on the moved-issue test: #154 is the simulation API's transferred issue, not this repository's. Three tests fail: the script still uses a per-version title. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update-policyengine-package.sh now reuses the oldest open issue titled "Update the PolicyEngine bundle", or opens one, instead of an issue per version. Most bundle-update PRs are closed without merging when a newer version arrives, and an issue per version stayed open for each of them; with one shared issue, the first PR to merge closes it and the next update opens a new one. The PR skill and pr-description.yml no longer say the master ruleset already requires the PR description check: they say the ruleset has to require it by name for it to block a merge. The skill also says that a bundle-update PR opened before the rule needs its description edited by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tracking issue shared by every bundle-update PR is closed by the first one to merge, so the other open ones name a closed issue and fail the PR description check on their next push (#3918 merged while #3925 was open, and #3953 while #3954 was). Go back to one issue per version, and test a new script that closes a bundle-update PR's issue when the PR is closed without merging, so superseded updates leave no issue open: - it closes the open issue titled for the branch's version that the first line of the description names, with reason "not planned"; - it leaves merged PRs, other branches, descriptions without a Fixes first line, and issues with another title, closed or a pull request alone; - its workflow runs on closed pull requests, only for unmerged bundle-update branches from this repository, with issues: write. Also test that the bundle-update script resolves the issue after the lock and before the push, so a failed lookup pushes no branch, and that it says which lookup failed. 25 cases fail: the script and workflow don't exist, and the bundle-update script uses the shared title and looks up the issue after pushing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update-policyengine-package.sh goes back to one tracking issue per version, titled "Update PolicyEngine bundle to <version>". A shared issue was closed by the first bundle-update PR to merge, which left the other open ones naming a closed issue. - New workflow close-bundle-update-issue.yml runs on closed pull requests from auto/update-policyengine-bundle-* branches in this repository that weren't merged, and runs close_bundle_update_issue.sh with the job token (issues: write). The script closes the issue the description's first line names, as "not planned", only when it is open and titled for the branch's version, so superseded updates leave no issue open. - The bundle-update script resolves the issue after the lock and the version check and before it creates and pushes the branch, so a failed lookup no longer leaves a pushed branch without a PR, and it says which issue call failed. - The PR skill describes both, and the rule's scope lists the new files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- GitHub doesn't run pull_request workflows while a pull request has a merge conflict, and a superseded bundle update usually conflicts with master: 14 of the 28 closed without merging since 2026-10-01 did when they were closed. Test that close-bundle-update-issue.yml listens for pull_request_target, checks out only the default branch's script, and can read pull requests, which gh issue close needs to look the number up. - Test that the bundle-update script closes the issue it opened when its push is rejected (as when a concurrent run for the version pushed first) or gh pr create fails, unless an open pull request for the branch names the issue, and never closes an issue it reused. - Test that the PR description check's message gives the skill's path and anchor as one link. 6 cases fail against the current workflow and scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- close-bundle-update-issue.yml now runs on pull_request_target, which GitHub runs even when the pull request conflicts with master, as most superseded bundle updates do; pull_request workflows don't run then. It uses the default branch's workflow and script, checks out nothing from the pull request, and reads only the event payload. Its token can also read pull requests, which gh issue close needs to look the number up. - update-policyengine-package.sh closes the tracking issue it opened when its push is rejected or gh pr create fails, unless an open pull request for the branch names it, so a concurrent run for the same version or a failed run leaves no issue that no pull request names. It never closes an issue it reused. - The PR description check's message gives the skill's path and anchor as one link. - The PR skill says to reopen a bundle-update PR's issue along with the PR, and what a PR into a branch that predates the check sees. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk
marked this pull request as ready for review
November 4, 2026 15:28
This was referenced Nov 4, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4173
Summary
Ports the simulation API's PR description check (PolicyEngine/policyengine-sim-api#862) and enforces
api.process.pr-tracking-issue, which wasdoc-only..github/workflows/pr-description.yml, jobPR description. It runs on everyopened,edited,reopenedandsynchronizeevent of an open pull request and fails unless the description's first line is exactlyFixes #Nandgh api repos/<this repository>/issues/N, with the job's token, reports an open issue in this repository. The workflow's permissions arecontents,issuesandpull-requestsread. It has no branches or paths filter and no stack condition, so every open pull request in a stack runs it, as in the simulation API. It is not part ofpr.yml, soCI completedoesn't need it; it becomes a separately required check onmasterin a follow-up ruleset change..github/scripts/check_pr_description.py, the simulation API's script with this repository's rule id. It uses only the standard library andgh, so the job installs nothing.Fixes #N. Every PolicyEngine bundle-update PR so far (for example Update PolicyEngine bundle to 6.13.0 #4169) started with## Summary.update-policyengine-package.shnow reuses the oldest open issue titled "Update PolicyEngine bundle to " or opens one, after the lock and version checks and before it pushes the branch, on both paths that open a PR. A failed issue lookup therefore pushes no branch, and the script says which issue call failed. When the push is rejected (as when a concurrent run for the same version pushed first) orgh pr createfails after the run opened the issue, the script closes that issue again, unless an open PR for the branch names it; it never closes an issue it reused. The daily relock PR (open_relock_pull_request.sh) already starts withFixes #N. No other workflow or script opens pull requests..github/workflows/close-bundle-update-issue.ymland script.github/scripts/close_bundle_update_issue.sh. Merging a bundle-update PR closes its issue through theFixesline, but most are closed without merging when a newer version arrives: 27 of the 32 closed since 2026-10-01 (the simulation API, which also opens an issue per version, has 17 such issues still open). On aclosedevent for an unmerged PR from anauto/update-policyengine-bundle-*branch in this repository, the job closes the issue the description's first line names, as "not planned", but only when it is open and titled for the branch's version. It runs onpull_request_target, because GitHub doesn't runpull_requestworkflows while a PR has a merge conflict, and 14 of those 28 superseded PRs conflicted with master when they were closed.pull_request_targetruns the default branch's workflow and script; the job checks out nothing from the PR and reads only the event payload, and its condition skips PRs from other repositories. One issue per version, rather than one shared by every bundle-update PR, means no open PR names an issue that another PR's merge closed (as Update PolicyEngine bundle to 6.3.4 #3918 merged while Update PolicyEngine bundle to 6.4.0 #3925 was open).api.process.pr-tracking-issueisenforcedas aworkflow-job, with proof fixtures;pr-description.ymlis added to[discovery].workflows.docs/generated/rules.mdandAGENTS.mdare re-rendered..github/**may now run other programs (S603, S607), asscripts/**already may, because the check callsgh.api.quality.ci-cd-scripts-ruff-exemptionslists the two codes. No inline suppressions are added.Fixtures
rules/fixtures/api.process.pr-tracking-issue/holds fourpull_requestevent payloads:bad_event.json: bundle-update PR Update PolicyEngine bundle to 6.13.0 #4169's description, with noFixesline;good_event.json: the same description withFixes #4170first, as the fixed script writes it;bad_fixes_not_first_event.json: PR Route API v1 simulations through the Stage 5 entrypoint #3776's description, shortened, withFixes #3775on line 10;bad_other_repository_event.json: a description whose first line names an issue in another repository.Environment variables
GH_TOKEN(PR description job)ghincheck_pr_description.pygithub.token, set in the step'senv:; the workflow'spermissions:limit it to read.GITHUB_EVENT_PATH,GITHUB_REPOSITORYcheck_pr_description.pyGH_TOKEN(bundle update)ghinupdate-policyengine-package.shsteps.app-token.outputs.tokenfromactions/create-github-app-token, built from the organisation secretsAPP_IDandAPP_PRIVATE_KEY(visible to all repositories). The relock workflow creates its tracking issues with a token built the same way (issue #4100 was opened by the PolicyEngine App), so the token can create issues.GITHUB_REPOSITORY(bundle update)update-policyengine-package.shPolicyEngine/policyengine-api(the job only runs in that repository).GH_TOKEN(closing job)ghinclose_bundle_update_issue.shgithub.token, set in the step'senv:, withissues: write,pull-requests: readandcontents: readfrom the workflow'spermissions:.telemetry-error-issues.ymlalready writes issues withgithub.tokenandissues: write(for example #4170, opened byapp/github-actions).GITHUB_EVENT_PATH,GITHUB_REPOSITORY(closing job)close_bundle_update_issue.shTests
Red-green, run by node ID, in four pairs of commits, each a failing-test commit and then the change that makes it pass: the check itself (59 cases failed against
master), a shared bundle issue from review round 1 (3 failed), per-version issues with the closing workflow from review round 2 (25 failed), andpull_request_targetwith the cleanup of unused issues from review round 3 (6 failed). The shared issue was replaced in round 2. All 97 cases in the three files pass:tests/unit/test_pr_description_check.py: first-line parsing (a valid description, noFixesline, the line not first, a blank first line, leading spaces, CRLF, other wording, another repository's issue), the issue lookup through a fakegh(open, closed, a pull request, missing, moved to another repository, a failed lookup), the workflow's triggers, condition, permissions and step, and the proof on the four fixtures.tests/unit/test_bundle_update_pins.py: the tracking issue is opened, or the oldest open one titled for the version is reused (issues with other titles are not), in the runner'sGITHUB_REPOSITORY, after the lock and before the push, and the PR description starts withFixes #N, for a new branch and for an existing branch without a PR; a failed issue lookup pushes nothing and opens neither an issue nor a PR; a rejected push or a failed PR closes an issue the run opened, but not one it reused or one an open PR names. The fakeghapplies the script's--jqexpression withjq. The file's seven existing tests, which use the changed fake, also pass.tests/unit/test_close_bundle_update_issue_script.py: the script closes the issue an unmerged bundle-update PR names; it leaves merged PRs, other branches, descriptions without aFixesfirst line, and issues with another title, already closed or a PR alone; it fails when the lookup fails; and the workflow'spull_request_targettrigger, condition, permissions, and a checkout of the default branch's script only.python -m scripts.rules guard --base-ref origin/master,python scripts/run_quality_guards.py,ruff format --check .,ruff check .,uv lock --checkand the changelog fragment check pass locally.Open pull requests without a
FixeslineRunning the check on every open PR's description found five that fail: the bundle-update PRs #4169 and #4163 (
## Summary), #4166 (Tracking issue: none yet.), and #3834 and #3510 (prose first lines). Each fails the new check on its next edit or push, and rerunning the bundle workflow doesn't fix #4169 or #4163, because the script skips a version whose PR is open: their descriptions need editing. Once the check is required, every open PR, including those that comply, showsPR descriptionas expected but not reported until its next edit, push or reopen. Four open PRs target branches other thanmaster(#4132, #4020, #4014, #4010); those branches don't contain the check script, so the job fails on them withcan't open fileuntil master is merged into the branch or the PR is retargeted tomaster. The ruleset applies only tomaster, so this doesn't block merges into those branches.Known limitations
🤖 Generated with Claude Code