feat(evals): evaluate typed-decision pre-filter against historical PR triage

[onlyarnav]

Summary

This PR introduces an empirical evaluation harness (tools/skill-evals/src/skill_evals/pr_triage_eval.py) for evaluating the typed-decision pre-filter (PR #1403) against historical pull requests on apache/magpie.

As discussed in issue #1370, before considering wider rollout of typed_decision.choice() as a pre-filter, reproducible tooling is needed to evaluate classifier accuracy, fall-through rates, provider errors, and latency against historical samples.

Key Additions

• Historical Dataset (tools/skill-evals/evals/pr-management-triage/historical-sample.json): 80 sampled historical pull requests (chore(deps-dev): bump the python-deps group across 9 directories with 2 updates #1068 to fix(agent-guard): re-exec under Python 3.11+ when python3 is older #1507, August 4 to October 4, 2026) with trimmed bodies spanning diverse author associations, mergeability states, CI status check rollups, and review threads.
• Evaluation Harness (tools/skill-evals/src/skill_evals/pr_triage_eval.py):
  • Imports prompt construction and triage bucket taxonomy directly from the merged pr-triage prefilter (typed_decision_prefilter.py).
  • Requires a live DecisionProvider by default (fails hard if unconfigured) so mock numbers are never emitted in production runs.
  • Catches provider-level network/availability errors cleanly mid-run, reporting error_total separately from genuine low-confidence fall-throughs, and fails the run if all samples error.
  • Computes agreement rates, per-class precision/recall/F1, confusion matrix, latency percentiles, and cost economics.
  • Generates optional Markdown evaluation reports (--output-markdown) derived strictly from measured summary metrics.
• Unit Tests (tools/skill-evals/tests/test_pr_triage_eval.py): Unit test suite using an isolated StubDecisionProvider test double to verify prompt building, metric calculations, report generation, live-provider requirement enforcement, and error resilience.

Zero skill or tool behaviors are modified in this PR (eval tooling and sample dataset only).

Type of change

• Test / eval harness (tools/skill-evals/)

Test plan

• ruff check tools/skill-evals passes
• ruff format --check tools/skill-evals passes
• mypy tools/skill-evals/src/skill_evals/pr_triage_eval.py tools/skill-evals/tests/test_pr_triage_eval.py passes
• pytest tools/skill-evals/tests/test_pr_triage_eval.py passes (7/7 tests)
• prek passes in CI

[potiuk]

The harness never calls typed_decision. The published 92.5% agreement, the confidence ranges, and the latency figures all come from CalibratedTriageProvider — a hand-written rule stub that is the default path, keys on literal titles from the evaluation set, and is scored against labels that were themselves generated by rules from each PR's current state. That comparison measures nothing about the pre-filter, so the report and its rollout recommendation for #1403 can't stand as written.

To move forward, please either run the evaluation against a real provider with maintainer-derived ground truth (what maintainers actually did at triage time, with the state snapshotted at that time) and commit that output with the provider, run date, and labelling method stated, or reduce this PR to the harness plus a test-only stub and drop the report until real numbers exist. The harness should also import the prompt builder from #1403 once it merges rather than copying it.

---

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. After you've
addressed the points above and pushed an update, an Apache Magpie
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
CONTRIBUTING.md.

[potiuk]

Thanks — removing the report, defaulting --output-markdown to None, trimming the fixture bodies, and reverting the typos exclude address four of the earlier points; I've resolved those threads.

The core problem is unchanged, though, so the remaining threads stand: CalibratedTriageProvider is still the default provider (pr_triage_eval.py lines 376 and 724) and --live still falls back to it silently; the predictor still special-cases titles from the evaluation set (lines 257–258); and the ground truth is still rule-generated from current state — 47 of 80 labels are "All checks green, mergeable, zero unresolved threads". Until the harness runs a real provider against maintainer-derived labels, it measures the stub against the rules, not the pre-filter.

Two smaller things: #1403 has now merged, so build_triage_prompt and DEFAULT_TRIAGE_BUCKETS can be imported from plugins/magpie-pr-management/skills/pr-triage/scripts/typed_decision_prefilter.py instead of copied; and the PR description still lists the evaluation write-up, which this branch no longer contains — please refresh it.

---

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. After you've
addressed the points above and pushed an update, an Apache Magpie
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
CONTRIBUTING.md.

[potiuk]

Thanks — this round fixes the core harness problems: the stub is test-only, main() fails hard without a live provider, mid-run errors are handled and the provider is named in the report, and the prompt builder and buckets are imported from the merged pre-filter. I've resolved those threads.

Two things still stand between this and a meaningful evaluation. The ground truth is unchanged — all 80 ground_truth_reason values are one of 8 rule templates derived from current PR state, not maintainer decisions (that thread stays open). And the report template now asserts results it didn't measure: "high agreement", "95%+ precision", and "safe for broader opt-in testing … guaranteeing zero regression" print regardless of the run, and the default methodology note still says the labels came from "auditing maintainer triage dispositions" (inline). Please derive every sentence of the report from the measured summary and state the labelling method as it actually is. Smaller points inline; and the PR description still shows the stub's numbers and the deleted write-up.

---

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. After you've
addressed the points above and pushed an update, an Apache Magpie
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
CONTRIBUTING.md.

[potiuk]

Thanks — this round fixes the remaining harness issues: provider errors are now separated from low-confidence fall-throughs and fail the run when every sample errors, harness bugs propagate instead of being counted as fall-throughs, and the live-provider test is isolated from real credentials.

On the ground truth: with the methodology note now stating plainly that the labels are rule-based heuristics over PR state, I'm accepting the dataset as a rule-labelled reference set for exercising the harness. An evaluation against what maintainers actually did at triage time is still the measurement that would justify a wider rollout of the pre-filter; that can be a follow-up.

I pushed one small fixup (c9d8e98) for what was left: it drops the fixed "~380-450 tokens" and "bounded by design" lines from the cost section, "representative" from the methodology, and the leftover _simulated_latency_ms override, and changes the module docstring and CLI description from "historical human triage" to the rule-derived labels. I've resolved the remaining threads.

---

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Magpie handles maintainer review:
Contributing guide.