Skip to content

Check that every PR description starts with Fixes #N for an open issue - #4177

Merged
anth-volk merged 8 commits into
masterfrom
pr-description-check
Nov 4, 2026
Merged

anth-volk merged 8 commits into
masterfrom
pr-description-check

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4173

Summary

Ports the simulation API's PR description check (PolicyEngine/policyengine-sim-api#862) and enforces api.process.pr-tracking-issue, which was doc-only.

  • New workflow .github/workflows/pr-description.yml, job PR description. It runs on every opened, edited, reopened and synchronize event of an open pull request and fails unless the description's first line is exactly Fixes #N and gh api repos/<this repository>/issues/N, with the job's token, reports an open issue in this repository. The workflow's permissions are contents, issues and pull-requests read. It has no branches or paths filter and no stack condition, so every open pull request in a stack runs it, as in the simulation API. It is not part of pr.yml, so CI complete doesn't need it; it becomes a separately required check on master in a follow-up ruleset change.
  • New script .github/scripts/check_pr_description.py, the simulation API's script with this repository's rule id. It uses only the standard library and gh, so the job installs nothing.
  • Bundle-update PRs now start with Fixes #N. Every PolicyEngine bundle-update PR so far (for example Update PolicyEngine bundle to 6.13.0 #4169) started with ## Summary. update-policyengine-package.sh now reuses the oldest open issue titled "Update PolicyEngine bundle to " or opens one, after the lock and version checks and before it pushes the branch, on both paths that open a PR. A failed issue lookup therefore pushes no branch, and the script says which issue call failed. When the push is rejected (as when a concurrent run for the same version pushed first) or gh pr create fails after the run opened the issue, the script closes that issue again, unless an open PR for the branch names it; it never closes an issue it reused. The daily relock PR (open_relock_pull_request.sh) already starts with Fixes #N. No other workflow or script opens pull requests.
  • New workflow .github/workflows/close-bundle-update-issue.yml and script .github/scripts/close_bundle_update_issue.sh. Merging a bundle-update PR closes its issue through the Fixes line, but most are closed without merging when a newer version arrives: 27 of the 32 closed since 2026-10-01 (the simulation API, which also opens an issue per version, has 17 such issues still open). On a closed event for an unmerged PR from an auto/update-policyengine-bundle-* branch in this repository, the job closes the issue the description's first line names, as "not planned", but only when it is open and titled for the branch's version. It runs on pull_request_target, because GitHub doesn't run pull_request workflows while a PR has a merge conflict, and 14 of those 28 superseded PRs conflicted with master when they were closed. pull_request_target runs the default branch's workflow and script; the job checks out nothing from the PR and reads only the event payload, and its condition skips PRs from other repositories. One issue per version, rather than one shared by every bundle-update PR, means no open PR names an issue that another PR's merge closed (as Update PolicyEngine bundle to 6.3.4 #3918 merged while Update PolicyEngine bundle to 6.4.0 #3925 was open).
  • Registry: api.process.pr-tracking-issue is enforced as a workflow-job, with proof fixtures; pr-description.yml is added to [discovery].workflows. docs/generated/rules.md and AGENTS.md are re-rendered.
  • Ruff: .github/** may now run other programs (S603, S607), as scripts/** already may, because the check calls gh. api.quality.ci-cd-scripts-ruff-exemptions lists the two codes. No inline suppressions are added.
  • Docs: the PR skill's "Tracking Issues" section describes the check, its failures, how to fix them, and how automated PRs open and close their issues; "Stacked PRs and CI" mentions the job and says the master ruleset has to require it by name for it to block a merge (the follow-up ruleset change does that).

Fixtures

rules/fixtures/api.process.pr-tracking-issue/ holds four pull_request event payloads:

Environment variables

Variable Read by Provider and wiring
GH_TOKEN (PR description job) gh in check_pr_description.py The job's automatic github.token, set in the step's env:; the workflow's permissions: limit it to read.
GITHUB_EVENT_PATH, GITHUB_REPOSITORY check_pr_description.py Set by the GitHub Actions runner for every step.
GH_TOKEN (bundle update) gh in update-policyengine-package.sh Unchanged: steps.app-token.outputs.token from actions/create-github-app-token, built from the organisation secrets APP_ID and APP_PRIVATE_KEY (visible to all repositories). The relock workflow creates its tracking issues with a token built the same way (issue #4100 was opened by the PolicyEngine App), so the token can create issues.
GITHUB_REPOSITORY (bundle update) update-policyengine-package.sh Set by the runner; the script falls back to PolicyEngine/policyengine-api (the job only runs in that repository).
GH_TOKEN (closing job) gh in close_bundle_update_issue.sh The job's automatic github.token, set in the step's env:, with issues: write, pull-requests: read and contents: read from the workflow's permissions:. telemetry-error-issues.yml already writes issues with github.token and issues: write (for example #4170, opened by app/github-actions).
GITHUB_EVENT_PATH, GITHUB_REPOSITORY (closing job) close_bundle_update_issue.sh Set by the GitHub Actions runner.

Tests

Red-green, run by node ID, in four pairs of commits, each a failing-test commit and then the change that makes it pass: the check itself (59 cases failed against master), a shared bundle issue from review round 1 (3 failed), per-version issues with the closing workflow from review round 2 (25 failed), and pull_request_target with the cleanup of unused issues from review round 3 (6 failed). The shared issue was replaced in round 2. All 97 cases in the three files pass:

  • tests/unit/test_pr_description_check.py: first-line parsing (a valid description, no Fixes line, the line not first, a blank first line, leading spaces, CRLF, other wording, another repository's issue), the issue lookup through a fake gh (open, closed, a pull request, missing, moved to another repository, a failed lookup), the workflow's triggers, condition, permissions and step, and the proof on the four fixtures.
  • tests/unit/test_bundle_update_pins.py: the tracking issue is opened, or the oldest open one titled for the version is reused (issues with other titles are not), in the runner's GITHUB_REPOSITORY, after the lock and before the push, and the PR description starts with Fixes #N, for a new branch and for an existing branch without a PR; a failed issue lookup pushes nothing and opens neither an issue nor a PR; a rejected push or a failed PR closes an issue the run opened, but not one it reused or one an open PR names. The fake gh applies the script's --jq expression with jq. The file's seven existing tests, which use the changed fake, also pass.
  • tests/unit/test_close_bundle_update_issue_script.py: the script closes the issue an unmerged bundle-update PR names; it leaves merged PRs, other branches, descriptions without a Fixes first line, and issues with another title, already closed or a PR alone; it fails when the lookup fails; and the workflow's pull_request_target trigger, condition, permissions, and a checkout of the default branch's script only.

python -m scripts.rules guard --base-ref origin/master, python scripts/run_quality_guards.py, ruff format --check ., ruff check ., uv lock --check and the changelog fragment check pass locally.

Open pull requests without a Fixes line

Running the check on every open PR's description found five that fail: the bundle-update PRs #4169 and #4163 (## Summary), #4166 (Tracking issue: none yet.), and #3834 and #3510 (prose first lines). Each fails the new check on its next edit or push, and rerunning the bundle workflow doesn't fix #4169 or #4163, because the script skips a version whose PR is open: their descriptions need editing. Once the check is required, every open PR, including those that comply, shows PR description as expected but not reported until its next edit, push or reopen. Four open PRs target branches other than master (#4132, #4020, #4014, #4010); those branches don't contain the check script, so the job fails on them with can't open file until master is merged into the branch or the PR is retargeted to master. The ruleset applies only to master, so this doesn't block merges into those branches.

Known limitations

  • Reopening a bundle-update PR that the closing job handled leaves its issue closed, so the check fails until the issue is reopened; the skill says to reopen both. No PR among the last 100 was reopened.
  • A reopen within seconds of the close could pass the check before the closing job closes the issue.

🤖 Generated with Claude Code

anth-volk and others added 2 commits November 4, 2026 14:42
Port the simulation API's tests for check_pr_description.py and
pr-description.yml (rule api.process.pr-tracking-issue), with four event
payload fixtures: bundle-update PR #4169 without a Fixes line and fixed,
PR #3776 with its Fixes line not first, and a description naming an issue
in another repository. Test that the bundle-update script opens or reuses
a tracking issue and puts Fixes #N first in the PR it opens.

The check script and the workflow don't exist yet, and the bundle-update
script writes no Fixes line, so all 59 cases fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the simulation API's PR description check (PolicyEngine/policyengine-sim-api#862).
A new workflow, pr-description.yml, runs on every opened, edited,
reopened and synchronize event of an open pull request, with no stack
condition, and fails unless the description's first line is exactly
Fixes #N and gh api, with the job's read-only token, reports N as an
open issue in this repository.

- Enforce api.process.pr-tracking-issue with the new job, with proof
  fixtures, and add pr-description.yml to the registry's discovered
  workflows.
- The PolicyEngine bundle-update script now reuses the open "Update
  PolicyEngine bundle to <version>" issue or opens one, and starts the
  pull request's description with Fixes #N on both paths that open a
  pull request. The daily relock script already did.
- Let CI/CD scripts under .github run other programs (S603, S607), as
  command-line scripts under scripts/ already may.
- Describe the check in the PR skill, AGENTS.md and the README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.30%. Comparing base (dfcbf2e) to head (88a8a88).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #4177       +/-   ##
===========================================
+ Coverage   53.80%   88.30%   +34.49%     
===========================================
  Files         281      293       +12     
  Lines       16310    17506     +1196     
  Branches     2639     2875      +236     
===========================================
+ Hits         8776    15459     +6683     
+ Misses       6803     1335     -5468     
+ Partials      731      712       -19     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

anth-volk and others added 2 commits November 4, 2026 15:03
27 of the 32 bundle-update PRs closed since 2026-10-01 were closed without
merging, so an issue per version would be left open for each of them.
Test that every bundle-update PR names the oldest open issue titled
"Update the PolicyEngine bundle", or a new one, and that issues with other
titles, including the per-version title, are not reused.

- The fake gh now applies the script's --jq expression to a list of open
  issues with jq, so the tests exercise the selection and sort order.
- run_update no longer passes the shell's GITHUB_REPOSITORY to the script,
  which made a test fail wherever that variable names another repository;
  a test passes one explicitly to check the script uses it.
- Correct the comment on the moved-issue test: #154 is the simulation
  API's transferred issue, not this repository's.

Three tests fail: the script still uses a per-version title.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update-policyengine-package.sh now reuses the oldest open issue titled
"Update the PolicyEngine bundle", or opens one, instead of an issue per
version. Most bundle-update PRs are closed without merging when a newer
version arrives, and an issue per version stayed open for each of them;
with one shared issue, the first PR to merge closes it and the next update
opens a new one.

The PR skill and pr-description.yml no longer say the master ruleset
already requires the PR description check: they say the ruleset has to
require it by name for it to block a merge. The skill also says that a
bundle-update PR opened before the rule needs its description edited by
hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 2 commits November 4, 2026 15:20
A tracking issue shared by every bundle-update PR is closed by the first
one to merge, so the other open ones name a closed issue and fail the PR
description check on their next push (#3918 merged while #3925 was open,
and #3953 while #3954 was). Go back to one issue per version, and test a
new script that closes a bundle-update PR's issue when the PR is closed
without merging, so superseded updates leave no issue open:

- it closes the open issue titled for the branch's version that the first
  line of the description names, with reason "not planned";
- it leaves merged PRs, other branches, descriptions without a Fixes first
  line, and issues with another title, closed or a pull request alone;
- its workflow runs on closed pull requests, only for unmerged
  bundle-update branches from this repository, with issues: write.

Also test that the bundle-update script resolves the issue after the lock
and before the push, so a failed lookup pushes no branch, and that it says
which lookup failed.

25 cases fail: the script and workflow don't exist, and the bundle-update
script uses the shared title and looks up the issue after pushing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update-policyengine-package.sh goes back to one tracking issue per
version, titled "Update PolicyEngine bundle to <version>". A shared issue
was closed by the first bundle-update PR to merge, which left the other
open ones naming a closed issue.

- New workflow close-bundle-update-issue.yml runs on closed pull
  requests from auto/update-policyengine-bundle-* branches in this
  repository that weren't merged, and runs close_bundle_update_issue.sh
  with the job token (issues: write). The script closes the issue the
  description's first line names, as "not planned", only when it is open
  and titled for the branch's version, so superseded updates leave no
  issue open.
- The bundle-update script resolves the issue after the lock and the
  version check and before it creates and pushes the branch, so a failed
  lookup no longer leaves a pushed branch without a PR, and it says which
  issue call failed.
- The PR skill describes both, and the rule's scope lists the new files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 2 commits November 4, 2026 15:41
- GitHub doesn't run pull_request workflows while a pull request has a
  merge conflict, and a superseded bundle update usually conflicts with
  master: 14 of the 28 closed without merging since 2026-10-01 did when
  they were closed. Test that close-bundle-update-issue.yml listens for
  pull_request_target, checks out only the default branch's script, and
  can read pull requests, which gh issue close needs to look the number up.
- Test that the bundle-update script closes the issue it opened when its
  push is rejected (as when a concurrent run for the version pushed first)
  or gh pr create fails, unless an open pull request for the branch names
  the issue, and never closes an issue it reused.
- Test that the PR description check's message gives the skill's path and
  anchor as one link.

6 cases fail against the current workflow and scripts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- close-bundle-update-issue.yml now runs on pull_request_target, which
  GitHub runs even when the pull request conflicts with master, as most
  superseded bundle updates do; pull_request workflows don't run then.
  It uses the default branch's workflow and script, checks out nothing
  from the pull request, and reads only the event payload. Its token can
  also read pull requests, which gh issue close needs to look the number
  up.
- update-policyengine-package.sh closes the tracking issue it opened when
  its push is rejected or gh pr create fails, unless an open pull request
  for the branch names it, so a concurrent run for the same version or a
  failed run leaves no issue that no pull request names. It never closes
  an issue it reused.
- The PR description check's message gives the skill's path and anchor as
  one link.
- The PR skill says to reopen a bundle-update PR's issue along with the
  PR, and what a PR into a branch that predates the check sees.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 4, 2026 15:28
@anth-volk
anth-volk requested a review from a team as a code owner November 4, 2026 15:28
@anth-volk
anth-volk merged commit ed52f71 into master Nov 4, 2026
24 checks passed
@anth-volk
anth-volk deleted the pr-description-check branch November 4, 2026 15:28

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Check that every PR description starts with Fixes #N for an open issue

1 participant