Skip to content

Compare flags a task as disagreeing from its last attempt only, and shows no pass@k or pass^k #130

Closed

Description

@aitianshi

Priority

high

Components

  • app/nullius (backend)
  • app/web (routes)
  • templates and static files
  • docs and the site
  • catalogue (benchmarks, agents, capabilities)
  • the standard (ports, kits, relay, runs)
  • CI, workflows and skills

Evidence

The compare view decides that a task disagrees from one attempt per run. With repeated attempts, a task with the same pass count in both runs can be flagged as a flip, and a real change is not ranked above it.

In app/nullius/compare/comparison.py:

  • build() keeps one trial per run on each task row, overwriting earlier attempts: row[column] = trial; the other attempts are only counted in attempts[column].
  • TaskRow.differs compares the outcome (and rounded score) of those kept trials only.
  • Rows are sorted by (not row.differs, row.dataset, row.task): the size of the change plays no part.

The page says which attempt it shows (compare.html: "each task row and its detail show the last indexed attempt"), but the headline "N where they disagree" (n_differing) and the row-differs highlight are computed from that one attempt.

Reported by a reader of the compare view: "with repeated attempts a task passing 2 of 3 times in both runs can still look like a flip. Sorting by how far the pass count moved would probably put the real changes on top."

Screenshot

Two runs of one GSM8K problem, three attempts each (2 of 3 passed against 0 of 3): the row shows the last indexed attempts and "3 attempts · last indexed shown".

Compare task row: Resolved against Failed verification, 3 attempts, last indexed shown

Reproduction

  1. Take two jobs on the same dataset, each with 3 attempts per task.
  2. Find a task with 2 of 3 passing in both, where the last indexed attempt passed in one job and failed in the other.
  3. Open Compare on the two jobs: the task is highlighted as disagreeing and counted in "N where they disagree".
  4. A task at 0 of 3 in one job and 3 of 3 in the other sits in the same group, ordered by name.

Expected

A task disagrees when its pass rate (or mean score, for a scored benchmark) differs between the runs, not when its last attempts differ. The rows that moved most come first.

Acceptance

  • A task row carries, per run, every attempt in the cohort, read as the adjusted rate reads trials: the passes over the attempts that ran with infra attempts counted apart, and the mean score where scored.
  • differs and n_differing compare those per-run rates and how the attempts that ran ended: a task with the same pass count and the same endings in every run is not counted as disagreeing, and a single attempt that failed one way in one run and another way in the other still is.
  • Rows sort by how far the pass rate moved (for a scored task, the mean score), each rate read with one pass and one fail added so a single flipped attempt ranks below a steady gap, largest first; then the other rows whose rate moved; then the rest that disagree; then dataset and task.
  • Each cell shows the pass count as passes/attempts when a run has more than one attempt, infra attempts apart; the link to the attempt's trial remains.
  • A test builds two runs with repeated attempts and checks: 2/3 against 2/3 is not a disagreement, 0/3 against 3/3 sorts above 1/3 against 2/3, and the pass@k and pass^k values match the estimators.
  • When every run has two or more attempts per task, the run summaries show pass@k and pass^k for k from 1 to the smallest attempt count across eligible tasks, for Harbor and standard jobs alike, with the rules docs/field-guide.md#attempts states (a task with an infra attempt is left out; an agent's timeout or crash counts as a failure).
  • One implementation of the two estimators over (n, c) per task, used both by this view and by pass_hat in app/nullius/standard/job.py; its existing test still passes.
  • docs/field-guide.md and the compare page's wording describe disagreement by pass rate; the field contract is refreshed if a rendered field changes.

Dependencies and related

  • Existing pass^k / pass@k for standard runs: pass_hat in app/nullius/standard/job.py, written to result.json as pass_hat_k and shown nowhere in the app.

Activity

  1. self-assigned this
    on Nov 14, 2026
  2. added
    bugSomething isn't working
    P2Planned: limited in scope, has a way around it, or a missing feature
    on Nov 14, 2026
  3. added theissue type on Nov 14, 2026
  4. aitianshi commented on Nov 16, 2026

    @aitianshi
    CollaboratorAuthor

    Acceptance revised to what #146 builds, after a review of the implementation found three faults in the first rules:

    • pass@k left out every task with an errored attempt, an agent's timeout included. On a run of three tasks where the agent timed out on some attempts, pass^1 read 100% over one task while the adjusted rate was 56%. Now only a task with an infra attempt is left out, and a timeout or crash counts as a failure, so with no infra pass^1 equals the adjusted rate.
    • Judging a row by its pass rate alone stopped flagging a single attempt that failed verification in one run and timed out in the other. A row now also disagrees when the attempts that ran ended differently.
    • Ranking by the raw gap put one flipped attempt (1/1 against 0/1) above a steady gap (2/3 against 0/3). Each rate is now read with one pass and one fail added.

    Infra attempts (rate limits, provider and setup errors, the spending cap, cancellation) are counted apart from the pass count, as the adjusted rate counts them.

  5. changed the title [-]Compare flags a task as disagreeing from its last attempt only, and shows no pass@k or pass^k[/-] [+]Compare flags a task as disagreeing from its last repeat only, and shows no pass@k or pass^k[/+] on Nov 16, 2026
  6. changed the title [-]Compare flags a task as disagreeing from its last repeat only, and shows no pass@k or pass^k[/-] [+]Compare flags a task as disagreeing from its last attempt only, and shows no pass@k or pass^k[/+] on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Planned: limited in scope, has a way around it, or a missing featurebugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions