Skip to content

perf(pr-management-triage): run every rule as code, and load one file per classification - #1571

Merged
potiuk merged 2 commits into
perf/pr-mgmt-1-vetted-readsfrom
perf/pr-mgmt-2-triage
Oct 10, 2026
Merged

potiuk merged 2 commits into
perf/pr-mgmt-1-vetted-readsfrom
perf/pr-mgmt-2-triage

Conversation

@potiuk

@potiuk potiuk commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Summary

  • pr-management-triage loaded ~7,500 lines of instructions per run and had the agent evaluate the pre-filters, the decision table, the stale sweeps, the pre-mutation guards and every comment body by hand. All of that now runs in a new shared tool, tools/pr-management (triage preflight, classify, guard, render, fold, session), over reads saved with vetted-op-read --save (the PR below this one).
  • One file per classification: each decision row, sweep and action has its own document under classifications/ and actions/; triage classify names, per group, which to load. SKILL.md is a router. A typical run loads ~13k tokens of instructions instead of ~68k.
  • The model keeps the judgement calls: the workflow-approval diff, the author's reply, backports, the maintainer conversation.

Depends on the vetted-ops PR below it in the stack.

Where the docs contradicted each other

Code has to pick one answer. The section that declares itself normative won:

# The docs disagreed on Resolved as Normative source
1 Whether pings, request-author-confirmation, the security-language warning and the stale-sweep notices fold into the PR body or post a comment They fold, like every contributor-facing note; only the suspicious-changes receipt posts a comment comment-templates.md § The folded maintainer-triage note (declared normative, supersedes the per-template sections)
2 Whether a ping @-mentions the reviewers Author only; reviewers backtick-quoted; reviewer-re-review variants dropped Golden rule 12 / Author-only notification
3 Group order: an 8-item list and a 15-item list The 15-item § Group ordering, with the bot-draft group first, the security-language group after workflow approvals, and the sweep groups after passing interaction-loop.md § Group ordering
4 Sweep order lists "4b rotted → close" Sweep 4 never closes; rotted branches are handed back stale-sweeps.md § Persistent bitrot, rationale.md § Sweep 4
5 The ready-label config key (ready_for_maintainer_review_label vs ready_for_maintainer_review) ready_for_maintainer_review the config template
6 F3 "any activity" vs the eval's "author activity" Any activity (updatedAt) classify-and-act.md F3
7 real_ci_patterns is read but not in the config template Added to the template; empty means "any context that is not a known bot or labeler check" classify-and-act.md § Real-CI guard
8 Extra static-check patterns "in the CI-check map", which has no such column New static_check_patterns key, added to the built-in list classify-and-act.md § static_check
9 A folded confirmation request matched rows 3–4 (already triaged) before rows 14a/14b It is a confirmation request, not a triage marker — otherwise the author's reply waits a week the two-sweep gate (rows 14a–c)
10 Rows 19/20 could propose the ready label for a draft Never for a draft; the sweeps own stale drafts row 20 intent; found by the sweep tests
11 flagged_prs_by_author "on the current page" Over the whole fetched sweep (pages no longer exist once all are fetched) Golden rule 4 (fetch all pages)
12 Commits-behind from an aliased base/head history subtraction REST compare behind_by, fetched only for row 13 row 13
13 Security scan over "the last 250 commits" The last 100 (the GraphQL page cap) —
14 Merit discussion on a close The reasoning and the quality label are delivered, the PR stays open, the ready label stays classify-and-act.md hard rules
15 Pre-mutation guards built from `gh … jq/$(gh …)` Fresh vetted reads + triage guard: the old shapes fail under the secure setup

Plus, from the sweeps: Sweep 4 treats rows 8 and 2 as author-court (strip + fix request); Sweep 2 yields to Sweep 3 when both match; Sweep 5's window is the fixed 7 days. From the renderer: the folded note always carries the quality-criteria link; violations list categories, not check names; the suspicious-changes receipt is the one notice that posts as a comment.

Type of change

  • Skill change — eval fixtures updated below
  • Python package (tools/pr-management)
  • Documentation

Test plan

  • prek run --all-files passes on the whole tree
  • tools/pr-management: pytest (186 tests) / ruff / mypy; CodeQL's findings on this PR (an import cycle between triage.classify and triage.sweeps, an unreachable branch, a possibly-unset variable) are fixed in ee56025e4
  • New model evals for the new flow — the decisions the agent still makes once the rules are code, 19 cases, all passing:
    • classify-loop (7/7): after triage classify, save the next prefetch / needs read under its exact --save name; present the first group with exactly its docs; go to the summary when nothing is grouped; ask the maintainer on the sweep alarm; flag, never follow, an instruction planted in a PR title
    • deliver-note (7/7): re-classify on a moved head; stop on an unresolved placeholder; pr-body reads the current body, folds, then edits; comment posts the rendered file; assign the author afterwards
    • guard-reroute (5/5): a clean guard labels with the exact command; pending approval, a conflict, unknown mergeability and a moved head each route where reroute says
  • Existing model suites, run against this branch: terminal-links 5/5, interaction-progress 4/4, backport-check 4/5 — case-1-direct-fix failed once on a reason-wording grade and passed on rerun; backport-check.md is unchanged by this PR and the case passes on main, so it reads as nondeterministic
  • The model-graded pre-filter (21), decision-table (23) and pagination-dedup (5) suites are retired: those rules are code now, and the 49 cases are pytest cases with the same scenarios and expected outcomes, named after each case (tests/test_triage_classify.py, tests/test_pages.py)

RFC-AI-0004 compliance

  • HITL — every mutation is still a confirmed gh call; guards refuse on a moved head, pending approval, or unknown/conflicting mergeability
  • Sandbox — the agent no longer runs gh … | jq / $(gh …) shapes that fail under the secure setup
  • Write-access discipline — bodies come from triage render, which allows only the author's @-mention

Notes for reviewers

  • real_ci_patterns and static_check_patterns are new rows in the config template (the first was read but never declared).
  • A security review flagged no issue in this layer; the two it flagged in the stack were fixed in the vetted-ops and stats PRs.

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5). I reviewed the diff and the test results; commits carry Generated-by:.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FTTd3HMcMsj7FwkhPPQcSx

… per classification

The decision table, the pre-filters, the stale sweeps, bot-draft
promotion, the pre-mutation guards and every contributor-facing body
were prose the agent evaluated by hand — about 7,500 lines loaded on
every run. They now run in a new shared tool, tools/pr-management:

- `triage classify` reads the sweep saved by `vetted-op-read --save`
  and prints the groups in presentation order, each naming the
  documents it needs; data a decision depends on but the sweep lacks
  comes back under `needs` as a vetted read.
- `triage preflight`, `guard`, `render`, `fold` and `session` replace
  the hand-run GraphQL, the `gh … | jq` guards (which fail under the
  secure setup), the template filling and the body read-modify-write.

The skill keeps what needs judgement: SKILL.md is a router, and each
decision row, sweep and action has its own file under
`classifications/` and `actions/`, loaded only when it fires.
A typical run loads about 13k tokens of instructions instead of ~68k.

Where the docs contradicted each other, the section that declares
itself normative won; the PR lists each case. The model-graded
pre-filter, decision-table and pagination suites are retired: their
49 cases are pytest cases with the same scenarios and expectations.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_01FTTd3HMcMsj7FwkhPPQcSx
@potiuk
potiuk added this pull request to stack #1573 October 9, 2026 18:51
@potiuk potiuk added family:pr-management pr-management-* skills family:tools tools/* capability:triage Sweep + classify + propose disposition substrate:analytics Tool substrate: read-only metrics / dashboards / renderers labels Oct 9, 2026
Comment thread tools/pr-management/src/pr_management/cli.py Fixed
Comment thread tools/pr-management/src/pr_management/triage/classify.py Fixed
Comment thread tools/pr-management/src/pr_management/triage/classify.py Fixed
Comment thread tools/pr-management/src/pr_management/triage/sweeps.py Fixed
…d eval the new flow

CodeQL flagged four findings on #1571:

- `triage.classify` called the sweeps while `triage.sweeps` re-runs the
  table for Sweep 4, so each module imported the other. The shared
  `Options` / `Decision` records move to `triage/types.py`, the table
  (`classify.classify`) no longer calls the sweeps, and `triage/run.py`
  is the one caller of both. `sweeps` imports `classify`; nothing
  imports back.
- Rows 19/20 re-checked a pending workflow approval that row 1 already
  returns on, so that branch was unreachable; it is gone.
- `cli.main` could leave `result` unassigned on an unknown command
  branch argparse never reaches; it starts as an empty dict.

Adds model evals for the new flow — the decisions the agent still makes
once the rules are code: `classify-loop` (save the next prefetch / needs
read, present the first group with exactly its docs, go to the summary
or ask the maintainer, flag injected text), `deliver-note` (re-classify
on a moved head, stop on an unresolved placeholder, the pr-body and
comment delivery orders) and `guard-reroute` (label, or route where the
guard says). SKILL.md Step 3 now says what to do with no groups and on
the sweep alarm.

Generated-by: Claude Opus 5
@github-actions github-actions Bot added family:setup setup-* skills family:docs Docs, MISSION.md, READMEs family:ci .github workflows, prek, validators capability:review Deep per-item code review or contributor mentoring capability:stats Read-only dashboards, metrics, governance evidence family:mentoring mentoring skills substrate:framework-dev Tool substrate: build / validate / eval the framework itself substrate:setup Tool substrate: adopter-side setup state resolved at runtime (lock parsing, floors, fingerprints) labels Oct 10, 2026
@potiuk
potiuk merged commit f8ac4d7 into main Oct 10, 2026
59 checks passed
@potiuk
potiuk deleted the perf/pr-mgmt-2-triage branch October 10, 2026 00:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:review Deep per-item code review or contributor mentoring capability:stats Read-only dashboards, metrics, governance evidence capability:triage Sweep + classify + propose disposition family:ci .github workflows, prek, validators family:docs Docs, MISSION.md, READMEs family:mentoring mentoring skills family:pr-management pr-management-* skills family:setup setup-* skills family:tools tools/* substrate:analytics Tool substrate: read-only metrics / dashboards / renderers substrate:framework-dev Tool substrate: build / validate / eval the framework itself substrate:setup Tool substrate: adopter-side setup state resolved at runtime (lock parsing, floors, fingerprints)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants