Repository navigation
Description
Priority
high
Components
- app/nullius (backend)
- app/web (routes)
- templates and static files
- docs and the site
- catalogue (benchmarks, agents, capabilities)
- the standard (ports, kits, relay, runs)
- CI, workflows and skills
Evidence
The compare view decides that a task disagrees from one attempt per run. With repeated attempts, a task with the same pass count in both runs can be flagged as a flip, and a real change is not ranked above it.
In app/nullius/compare/comparison.py:
build()keeps one trial per run on each task row, overwriting earlier attempts:row[column] = trial; the other attempts are only counted inattempts[column].TaskRow.differscompares the outcome (and rounded score) of those kept trials only.- Rows are sorted by
(not row.differs, row.dataset, row.task): the size of the change plays no part.
The page says which attempt it shows (compare.html: "each task row and its detail show the last indexed attempt"), but the headline "N where they disagree" (n_differing) and the row-differs highlight are computed from that one attempt.
Reported by a reader of the compare view: "with repeated attempts a task passing 2 of 3 times in both runs can still look like a flip. Sorting by how far the pass count moved would probably put the real changes on top."
Screenshot
Two runs of one GSM8K problem, three attempts each (2 of 3 passed against 0 of 3): the row shows the last indexed attempts and "3 attempts · last indexed shown".
Reproduction
- Take two jobs on the same dataset, each with 3 attempts per task.
- Find a task with 2 of 3 passing in both, where the last indexed attempt passed in one job and failed in the other.
- Open Compare on the two jobs: the task is highlighted as disagreeing and counted in "N where they disagree".
- A task at 0 of 3 in one job and 3 of 3 in the other sits in the same group, ordered by name.
Expected
A task disagrees when its pass rate (or mean score, for a scored benchmark) differs between the runs, not when its last attempts differ. The rows that moved most come first.
Acceptance
- A task row carries, per run, every attempt in the cohort, read as the adjusted rate reads trials: the passes over the attempts that ran with infra attempts counted apart, and the mean score where scored.
-
differsandn_differingcompare those per-run rates and how the attempts that ran ended: a task with the same pass count and the same endings in every run is not counted as disagreeing, and a single attempt that failed one way in one run and another way in the other still is. - Rows sort by how far the pass rate moved (for a scored task, the mean score), each rate read with one pass and one fail added so a single flipped attempt ranks below a steady gap, largest first; then the other rows whose rate moved; then the rest that disagree; then dataset and task.
- Each cell shows the pass count as
passes/attemptswhen a run has more than one attempt, infra attempts apart; the link to the attempt's trial remains. - A test builds two runs with repeated attempts and checks: 2/3 against 2/3 is not a disagreement, 0/3 against 3/3 sorts above 1/3 against 2/3, and the pass@k and pass^k values match the estimators.
- When every run has two or more attempts per task, the run summaries show pass@k and pass^k for k from 1 to the smallest attempt count across eligible tasks, for Harbor and standard jobs alike, with the rules
docs/field-guide.md#attemptsstates (a task with an infra attempt is left out; an agent's timeout or crash counts as a failure). - One implementation of the two estimators over (n, c) per task, used both by this view and by
pass_hatinapp/nullius/standard/job.py; its existing test still passes. -
docs/field-guide.mdand the compare page's wording describe disagreement by pass rate; the field contract is refreshed if a rendered field changes.
Dependencies and related
- Existing pass^k / pass@k for standard runs:
pass_hatinapp/nullius/standard/job.py, written toresult.jsonaspass_hat_kand shown nowhere in the app.
Activity
aitianshi commentedon Nov 16, 2026
Acceptance revised to what #146 builds, after a review of the implementation found three faults in the first rules:
Infra attempts (rate limits, provider and setup errors, the spending cap, cancellation) are counted apart from the pass count, as the adjusted rate counts them.
[-]Compare flags a task as disagreeing from its last attempt only, and shows no pass@k or pass^k[/-][+]Compare flags a task as disagreeing from its last repeat only, and shows no pass@k or pass^k[/+][-]Compare flags a task as disagreeing from its last repeat only, and shows no pass@k or pass^k[/-][+]Compare flags a task as disagreeing from its last attempt only, and shows no pass@k or pass^k[/+]