[anth-volk]

Fixes #912

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

Stacked on #877. The rules guard (python -m scripts.rules guard, run in CI by scripts/run_quality_guards.py) compares rules/registry.toml with the base branch's registry. Before this PR it failed when a baseline count grew (a ratcheted rule's baseline is the file of violations it still tolerates) and when a rule's own status weakened without [rule.downgrade] reason. This PR closes the gaps that #877's type-checking guide listed, and makes every downgrade name an open tracking issue:

1. Packages that leave a rule. A package that an enforced or ratcheted Pyright rule lists on the base branch, and whose pyproject.toml still exists, must stay in a Pyright rule at least as strong: enforced for a package from an enforced rule, enforced or ratcheted for one from a ratcheted rule. Otherwise the guard fails, unless the rule that lists the package now has [rule.downgrade]. A package that no rule lists fails too. Moving a package to a rule at least as strong, or deleting it, passes.
2. New ratcheted rules. A rule the base branch doesn't have, that is ratcheted with a baseline count above 0, needs [rule.downgrade].
3. Tracking issues. [rule.downgrade] now needs issue, next to a non-empty reason. The loader always checks that issue is a positive whole number (TOML's true is rejected). On every run, the guard looks up every section's issue with gh api, through the PR description check's lookup, and fails unless it is an open issue (not a pull request) in issue_repository.
   • On GitHub Actions, the guard fails when neither GH_TOKEN nor GITHUB_TOKEN is set, when a lookup fails, when gh can't run, and when issue_repository isn't the repository the workflow runs in (GITHUB_REPOSITORY).
   • Outside GitHub Actions, without a token, it prints a note for each issue it didn't check. scripts/run_quality_guards.py now prints the rules guard's notes, as it already printed the ci-scripts guard's. With a token, it behaves as in CI.
4. CI. The quality-guards job in .github/workflows/pr.yml now has permissions: contents: read, issues: read. Only its "Run quality guards" step gets GH_TOKEN: ${{ github.token }}.
5. Docs. docs/engineering/skills/type-checking.md (the guard's checks, the "It doesn't check" list and the paragraph after it) and docs/engineering/skills/rules-registry.md (the [rule.downgrade] row, the Baselines bullets, and the commands paragraph) describe what the guard now checks, and what it still doesn't.

The registry has no [rule.downgrade] section on this branch, pyright-apis-integ or main, so nothing in the registry changes status. The only registry change is the new top-level issue_repository = "PolicyEngine/policyengine-sim-api". python3 -m scripts.rules render leaves docs/generated/rules.md and AGENTS.md unchanged.

Design choices

• Field names. [rule.downgrade] keeps reason and gains issue, the name [[rule.exception]] already uses for its tracking issue. The parsed section is Downgrade(reason, issue), which replaces Rule.downgrade_reason. A section without both is reported, and read as absent, so it never allows a weaker rule. Unknown keys in it are now reported.
• Which enforcer kinds the package comparison covers: Pyright only. It is the only kind with projects; the schema rejects that field on every other kind. A package that moves between two enforced rules keeps its check only if both run the same checker, so a comparison across kinds would be wrong anyway.
• Which [rule.downgrade] sections are looked up: all of them, on every run, not only those this pull request adds. The invariant is then "each downgrade names an open issue". The cost is that closing the issue while its section is still in the registry fails the guard on every pull request, as an expired review_by date does. So that restoring a rule doesn't cause this, the schema now rejects [rule.downgrade] on an enforced rule: the pull request that restores the rule must remove the section, and can then close the issue.
• Where the repository comes from. The new top-level issue_repository key means local runs and CI look issues up in the same repository; the existing repo key holds only the name. On GitHub Actions it must match the runner's GITHUB_REPOSITORY, which a pull request can't change.
• Reusing Check that every PR description starts with Fixes #N for an open issue #862's lookup. check_pr_description.py's lookup raised DescriptionError with messages that end in how to fix the description. require_open_issue now raises NotAnOpenIssueError (missing, moved or closed issue, or a pull request) or IssueLookupError (GitHub couldn't be asked, or its answer wasn't an issue), with messages that say only what is wrong. main() adds the same fixes as before. The guard loads the script by path and adds its own fixes. The PR description check's messages and exit codes are unchanged, except that the two lookup failures that said only "Re-run this job." now also say to check the job's GH_TOKEN, as the third already did.
• Locally, without a token. The guard counts GH_TOKEN or GITHUB_TOKEN as a token, since gh reads both. With neither, outside GitHub Actions, it skips the lookup and adds a note per issue: rule <id>: tracking issue #<n> was not checked, because neither GH_TOKEN nor GITHUB_TOKEN is set. CI checks it; to check it here, set one, for example GH_TOKEN="$(gh auth token)". With a token, a lookup that fails locally fails the guard, as in CI.

Commits

Each commit's tests were written first and run against the old code, or, where the module didn't exist yet, against a stub that returns no findings, so each test fails on its own assertion. The rules tool's tests: python3 -m unittest discover -s scripts/rules/tests -t ..

1. Require an issue number in every [rule.downgrade] (f731679b). Red: test_load couldn't import Downgrade; a reason-only downgrade still passed (test_downgrade_needs_reason_and_issue); the status fix text was the old one. That is 1 failure and 2 errors, out of 188 tests. Green: 195 tests pass.
2. Fail when a package leaves an enforced Pyright rule without a downgrade (608c6e45). Red, against a stub: 5 tests failed (6 failures; one test covers both ratcheted and advisory), and the 5 tests of allowed cases passed. Green: 205 tests pass.
3. Require a downgrade for a new rule that starts ratcheted (b0bd4f89). Red: 2 failures; a new ratcheted rule with 2 tolerated violations passed without a downgrade, and no note named the downgrade. The count-0 test passed. Green: 208 tests pass.
4. Separate the issue lookup from the PR description's fix text (a0882ae7). Red: 8 new in-process tests of require_open_issue failed with AttributeError, and the 52 existing PR-description tests passed. Green: 60 pass; the whole simulation-entry suite, 738 tests, passes.
5. Check that each [rule.downgrade] names an open tracking issue (8ae08ffc). Red, against a stub: 17 tests, 22 failures counting subtests. Most failed first on the loader rejecting issue_repository as an unknown key. A second red: a lookup script without require_open_issue crashed the guard with AttributeError. Green: 227 tests pass.
6. Give the quality guards a token that can read issues (94d57f7f). Red: 2 failures; the job had no permissions (KeyError), and no step got GH_TOKEN. Green: both pass, along with the 222 PR-description, workflow-script and deployment-asset tests, and the 10 stacked-CI tests.

Commits from the self-review:

7. Hold a ratcheted rule's packages to a rule at least as strong (29467523). Red: 2 failures; a package moving from a ratcheted rule to an advisory rule passed, and so did a package from a ratcheted rule that no rule lists. Green: 230 tests pass.
8. Reject true as an exception's issue number (201a2c90). Red: 1 subtest failure, issue = true. Green: 231 tests pass.
9. Name the token variable gh used when a lookup fails (67a51051). Red: 1 failure; with only GITHUB_TOKEN set, the fix said to check GH_TOKEN. Green: 232 tests pass.
10. Keep the tracking-issue schema tests away from the real gh (4e481bd5). Red: with a token set and a gh on PATH that records its calls, RegistrySchemaTest ran gh api repos/example/repo/issues/912. Green: the same run records no call, and the whole suite run that way records none.
11. Look tracking issues up only in the repository the workflow runs in (90e77bf5). Red: 1 failure; with GITHUB_REPOSITORY=example/other, the guard still looked the issue up in the registry's example/repo and passed. Green: 234 tests pass, both locally and with GITHUB_ACTIONS=true, GITHUB_REPOSITORY set and no token.
12. Say what the guard still doesn't check about downgrades (795c73af). Docs and one test comment only; 234 tests pass.

Suppressions added

None. The diff adds no # noqa, # type: ignore or # pyright: ignore. Where ruff objected, the code changed instead: run_guard's base comparison moved into _compare_with_base (C901, PLR0912), a constant was renamed (S105), and an annotation-only import moved under TYPE_CHECKING (TC001).

Real bugs found

• A ratcheted rule's packages could leave CI without a downgrade. A package could move from a ratcheted Pyright rule into a new advisory rule with no [[rule.ci]], and then leave the pyright job's matrix, with only notes. Fixed in commit 7.
• The pull request under review decided which repository the issue had to be in. Fixed in commit 11.
• [[rule.exception]] issue = true was accepted as issue 1 (this predates the branch). Fixed in commit 8.
• A rules test called the real gh whenever a token was set in the shell. It made a read-only request for example/repo, which the test then ignored. Fixed in commit 10.

Self-review

Self-review, not a review disposition. Two rounds, using the review-code format: the first by the agent that wrote the change, the second by a separate reviewer that was given the code and the plan but not the first round's findings.

Round 1 found:

• Bug: a ratcheted rule's packages weren't compared (commit 7).
• Bug: an exception's issue = true was accepted (commit 8).
• Code quality: a failed lookup's fix named only GH_TOKEN (commit 9).
• Code quality: the rules-registry guide's commands paragraph didn't mention gh (commit 9).

Round 2 found:

• Bug: issue_repository came from the head registry (commit 11).
• Code quality: test isolation (commit 10).
• Code quality, three docs statements the code didn't bear out (commit 12):
  • a move to a new ratcheted rule with a count above 0 still needs a downgrade;
  • removing [rule.downgrade] from a rule that stays weakened passes;
  • replacing a rule with a weaker one under a new id produces only notes.
• Not changed: typing the loaded lookup module with a Protocol. scripts/rules isn't type-checked, and the tests run against the real module, so a rename fails them.
• Not changed: under Python 3.11, test_pyright.PyrightCommandTest.test_list_packages_takes_no_other_pyright_options fails, because 3.11's argparse rejects the arguments before the command's own check runs. This branch doesn't touch that test or cli; under 3.13 and 3.14 all tests pass.
• It also confirmed that the PR description check's messages and exit codes are unchanged, apart from the retry text.

Round 3, by the same separate reviewer, covered commits 10–12 (67a51051..795c73af) and found no substantiated bugs. One minor note, not changed: a workflow that runs inside a fork, rather than a pull request from a fork into this repository, would fail whenever the registry has a [rule.downgrade], because its GITHUB_REPOSITORY names the fork. This repository runs CI only for same-repository branches, and forks have Actions turned off by default.

Gaps the guard still has, now listed in the guides:

• what a reason says, and whether the issue is about restoring this rule;
• which baseline entries changed;
• whether a [rule.downgrade] section is new, or was removed from a rule that stays weakened;
• a rule removed, or replaced under a new id, outside Pyright packages and new ratcheted rules.

Requiring a section to stay while its rule stays weakened would make every downgrade permanent-until-restored, so it is left as a decision for a person.

Environment variables

Variable	Read by	Provider	How it reaches the code
GH_TOKEN	gh, run by scripts/rules/tracking_issues.py through check_pr_description.require_open_issue; the guard checks it is set	github.token, the job's automatic token, limited by the quality-guards job's new permissions: contents: read, issues: read	Step-level env: on "Run quality guards". run_quality_guards.py runs the rules guard in its own process, so os.environ has it, and subprocess.run(["gh", ...]) passes no env=, so gh inherits it. The earlier "Test the rules registry tool" step doesn't get it.
GITHUB_TOKEN (environment variable)	the guard, as gh's second token variable	not set in CI: the runner doesn't export it, and no workflow sets it; locally, whatever the shell sets	—
GITHUB_ACTIONS	tracking_issues.check_tracking_issues (and the existing base-ref handling in guard.py)	the GitHub Actions runner, true in every step	runner environment
GITHUB_REPOSITORY	tracking_issues.check_tracking_issues (new); check_pr_description.main() (unchanged, only when run as the PR description job's script)	the GitHub Actions runner	runner environment
GITHUB_BASE_REF	baseline.default_base_ref (unchanged)	the runner, on pull_request events	runner environment
PATH (for gh)	subprocess.run(["gh", ...])	GitHub-hosted ubuntu-latest image	gh is preinstalled; the PR description job already relies on it, and passed on this PR

The tests set their own FAKE_GH_CALLS, FAKE_GH_ANSWERS and FAKE_CALLS_LOG, and clear GH_TOKEN, GITHUB_TOKEN, GITHUB_ACTIONS and GITHUB_REPOSITORY where they matter.

Not verified in CI:

• This PR is above the bottom of stack #867, so pr.yml's jobs are skipped here (CI complete fails by design). quality-guards will first run with these changes once Fix apis-integ's Pyright errors and enforce every Pyright rule #877 merges.
• Because the registry has no [rule.downgrade] section, that run won't call the lookup, so the token's live use is unexercised.
• The new job-level permissions drop every scope but contents and issues. Other jobs here already run actions/checkout, setup-python@v6 and setup-uv@v8.1.0 with only contents: read (publish-clients.yml, simulation-deploy.reusable.yml), so I expect this job to work, but it hasn't run yet.

Testing

Run before each push, from the repository root:

• python3 -m unittest discover -s scripts/rules/tests -t .: 234 tests pass. They also pass with GITHUB_ACTIONS=true, GITHUB_REPOSITORY set and no token, as in the CI step, and with tokens set and a recording gh first on PATH, which records no call.
• python3 scripts/run_quality_guards.py: every guard passes. python3 -m scripts.rules guard --base-ref origin/pyright-apis-integ and --base-ref origin/main pass.
• Ruff 0.15.22 (run with uv tool run ruff@0.15.22, because the local ruff is 0.16.8): format --check projects/*/src libs/*/src and check . pass, and so does format --check scripts .github/scripts.
• python3 -m scripts.rules pyright --sync: every package passes with no Pyright errors. Pyright doesn't check scripts/ or .github/scripts/, so its results can't change here. For 4e481bd5, a test-only change, I ran Pyright straight after pushing rather than before.
• simulation-entry, with uv run --frozen pytest: tests/test_pr_description_check.py (60) and tests/test_quality_guards_workflow.py (2) pass. The whole suite (738) passed after commit 4. simulation-executor's tests/test_ci_complete_gate.py (10) passes.

🤖 Generated with Claude Code