Skip to content

feat(runs): read a task's repeated attempts together, by pass rate, with pass@k and pass^k - #146

Merged
aitianshi merged 6 commits into
mainfrom
feat/compare-pass-count-sorting
Nov 17, 2026
Merged

aitianshi merged 6 commits into
mainfrom
feat/compare-pass-count-sorting

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #130
Closes #145

What

An attempt is one run of a task, counted whatever its result (Harbor's n_attempts, the standard's --repeat); a retry is a further try of an attempt after an infrastructure or agent failure, and only its last try is kept as the attempt's trial. A task's attempts in one job are now read together, by one backend module (nullius.runs.attempts), the way the run's adjusted rate reads trials: an infra attempt is counted apart and left out of the pass count, the means and pass@k, and an agent's timeout or crash counts as a failure. Attempts are numbered by the run's own number when its trials carry one (the standard's u0000r2), so an attempt retried after the others keeps its place, else by start time. The trial page gets a tab per attempt and an All attempts table. The job page shows one row per repeated task (its pass count, a square per attempt, the instruction score once, the mean tokens and times) that opens onto its attempts, and the run page shows pass@k and pass^k. Compare (#130) judges a task by its pass rate over every attempt instead of its last one, ranks the rows by how far that rate moved (read with one pass and one fail added, so a single flipped attempt ranks below a steady gap), shows pass@k and pass^k in its run summary, and gives each side of the side-by-side page its own attempt tabs. The standard's pass_hat calls the same estimators; what it writes is unchanged.

Touches

backend · routes · templates and static · docs and site · the standard (pass_hat calls the shared estimators)

Before and after

Before After
Job page: three rows named 0001, one per attempt Job page: one row for 0001, 2 of 3 passed, opened onto attempts 1 to 3 with the means on the task row
Run page rate block without pass@k Run page rate block with pass@k and pass^k for k 1 to 3
Trial page with no attempt tabs Trial page with 2/3 passed, a tab per attempt and All attempts
Compare run summary ending without pass@k Compare run summary with pass@1 to pass@3 and pass^1 to pass^3
Compare task row: 3 attempts, last indexed shown Compare task row: 2/3 passed against 0/3 passed

An infra attempt counted apart, after (a hand-made run in Harbor's layout): t1 reads 1/2 · 1 infra and its means leave the rate-limited attempt out; t2's timeout counts as a failure.

Job page: t1 at 1/2 with 1 infra, its rate-limited attempt beside a resolved and a failed one; t2 at 2/3 with a timeout

The screenshots are two real standard runs of GSM8K against a stand-in model server (one problem, three attempts each); the instruction scores are stand-in records.

Verified

  • On this branch merged with main (9b7c820): python3 app/tests/parallel.py, 2198 tests, OK; uv run --no-project --with-requirements app/requirements.txt python app/tests/parallel.py app/ui_tests, 542 tests, OK.
  • New tests: test_attempts.py (numbering, by the run's own number or by start; grouping, infra apart, the verdict rule, token means, the pass@k rules), test_comparison.py (disagreement by pass rate and by endings, single attempts, infra, ranking, a pass rate never set against a mean score, pass@k over the rows' tasks), test_attempt_routes.py and test_comparison_routes.py (tabs, All attempts, grouped rows with their means and infra, pass@k on the run page, the counts in Compare and side by side).
  • Field contract refreshed: attempts.Attempts is pinned. The docs-drift check's two findings (tau3-bench/search_relay, standard/pace) are on main too.
  • Browser checks in Chromium: job page, trial page, All attempts, Compare and side by side, dark and light at 1280 px and dark at 390 px, no page errors. The task row opens by click and by Enter; a filter opens the groups it matches and clearing it restores those opened by hand; sorting keeps a group together; hovering a closed task row lights its attempts on the board; every attempt's square has a colour, cancelled and scored included.
  • A real standard run whose attempt 2 was retried after attempt 3 lists attempts 1, 2, 3 in the run's own order.
  • Two review passes on the diff. What they found is fixed here: pass@k no longer leaves out a task where the agent timed out (it read pass^1 = 100% on a run whose adjusted rate is 56%), infra attempts are apart from every count, one flipped attempt no longer outranks a steady gap, and a single attempt that ended differently is still a disagreement.

Risk

  • Compare's "N where they disagree" and its row order change for every comparison, single attempts included: a row differs by pass rate or by how its attempts ended, and ranks by the gap read with one pass and one fail added.
  • pass@k and pass^k are new numbers on the run page and in Compare; a wrong rule would mislead. docs/field-guide.md#attempts states the rules, and the tests pin them.
  • A repeated task's attempt rows are hidden until opened; if board.js fails they stay hidden, and the trial tabs and All attempts still list them.
  • The standard's pass_hat runs in write_result at the end of every standard run, so an error in it would leave that run without its result.json. It now calls the shared estimators (imported inside the function); test_pass_hat_k_as_tau_bench_counts_it pins its output, which is unchanged, and the shard tests (test_standard_shard.py) write their results through it.
  • Undo: revert the squash commit. Nothing is migrated or written.

…ith pass@k and pass^k

Trials of one task in one job are its attempts, numbered by when they
started (nullius.runs.attempts). The trial page gets a tab per attempt and
an All attempts table; the job page shows one row per task with its pass
count, the instruction score once and the mean tokens and times; the run
page and Compare show pass@k and pass^k. Compare judges a task by its pass
rate over every attempt, ranks by how far it moved (read with one pass and
one fail added, so a single flipped attempt ranks below a steady gap), and
its side-by-side page gives each run its own attempt tabs.

Attempts are read as the adjusted rate reads trials: an infra attempt is
counted apart and left out of the pass count, the means and pass@k; an
agent's timeout or crash counts as a failure. The standard's pass_hat now
calls the same estimators; what it writes is unchanged.
…nt-sorting

# Conflicts:
#	app/static/style.css
@github-actions

This comment has been minimized.

A repeat is one run of a task, counted whatever its result (Harbor's
n_attempts, the standard's --repeat). An attempt is one try within a
repeat, taken again only when the infrastructure or the agent fails, and
only its last try is kept as the repeat's trial. The pages, Compare and its
command's JSON, the field guide and the module (nullius.runs.repeats) now
say repeat where they meant one. The standard still writes pass_hat_k's
`attempts` key, the name result.json has always carried.
@github-actions

This comment has been minimized.

@aitianshi aitianshi changed the title feat(runs): read a task's repeated attempts together, by pass rate, with pass@k and pass^k feat(runs): read a task's repeats together, by pass rate, with pass@k and pass^k Nov 16, 2026
…the run does

An attempt is one run of a task, counted whatever its result (Harbor's
n_attempts, the standard's --repeat); a retry is a further try of an
attempt after an infrastructure or agent failure, and only its last try is
kept as the attempt's trial. This takes back the word repeat for the first,
as Harbor names it, and the field guide's section says both.

A task's attempts are numbered by the run's own number when every trial
carries one (the standard's ..._u0042r2), so an attempt retried after the
others keeps its place; by start time otherwise, as Harbor's names carry no
number.
@github-actions

This comment has been minimized.

aitianshi added a commit that referenced this pull request Nov 16, 2026
…the agent, with --retries beside --attempts

Merges #146 and rebuilds this change in its words. An attempt is one run
of a unit, counted whatever its result: --attempts K, the older --repeat K
still read. A retry runs an attempt again after a failure a further try
may mend, by Harbor's retry rule (a rate limit, a provider, network or
setup error, an agent crash, an output-length or verifier error; never a
timeout, a refusal, a wrong model or key, a missing secret, a usage limit,
the spending cap, a stop, or a task the agent failed): --retries N, 0 by
default, so the single re-run of a rate-limited unit now needs --retries 1.

Earlier tries are kept in retried/<trial>.try-N, the kept trial records
its retries (standard.retries) and the config records retries. The builder
says Attempts per task and Retries per attempt for both runners, and a
Harbor recipe's retries default to 0. A retried attempt says so on the run
page, the trial page, All attempts and the live board, and the progress
counts every try. The driver protocol, the catalogue and the recipe formats
keep main's words.
@aitianshi aitianshi changed the title feat(runs): read a task's repeats together, by pass rate, with pass@k and pass^k feat(runs): read a task's repeated attempts together, by pass rate, with pass@k and pass^k Nov 16, 2026
…hanges a reading

Keep main's pass@k and pass^k text; add only the words for attempt and retry, which tasks the pages leave out, how a task's pass rate counts and when a Compare row disagrees.
aitianshi added a commit that referenced this pull request Nov 17, 2026
…empts

Brings in main through #146. The conflicts are resolved as in the rebased branch the suites passed on; the resolution also cuts the retries docs and the --retries help to the AGENTS.md documentation rule.
@github-actions

Copy link
Copy Markdown

🟡 PR judge · 6 of 8 yes · #146 ↔ #130

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/standard/job.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/standard/job.py
Question Verdict
Changes do what the issue asks · 8 of 8 criteria ✅ yes
Issue and pull request describe the same change ⚠️ partly
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ⚠️ partly
Before and after shown · 11 images · the diff touches what is visible ✅ yes
Acceptance criteria, 8 of 8 met
# Criterion Verdict
1 A task row carries, per run, every attempt in the cohort, read as the adjusted rate reads trials: the passes over the a… ✅ met (cut)
2 differs and n_differing compare those per-run rates and how the attempts that ran ended: a task with the same pass … ✅ met (cut)
3 Rows sort by how far the pass rate moved (for a scored task, the mean score), each rate read with one pass and one fail… ✅ met (cut)
4 Each cell shows the pass count as passes/attempts when a run has more than one attempt, infra attempts apart; the lin… ✅ met (cut)
5 A test builds two runs with repeated attempts and checks: 2/3 against 2/3 is not a disagreement, 0/3 against 3/3 sorts … ✅ met (cut)
6 When every run has two or more attempts per task, the run summaries show pass@k and pass^k for k from 1 to the smallest… ✅ met (cut)
7 One implementation of the two estimators over (n, c) per task, used both by this view and by pass_hat in `app/nullius… ✅ met (cut)
8 docs/field-guide.md and the compare page's wording describe disagreement by pass rate; the field contract is refreshe… ✅ met (cut)
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@aitianshi
aitianshi merged commit fe6f0a7 into main Nov 17, 2026
2 checks passed
@aitianshi
aitianshi deleted the feat/compare-pass-count-sorting branch November 17, 2026 22:19
aitianshi added a commit that referenced this pull request Nov 17, 2026
…the agent, with --retries beside --attempts (#154)

Closes #152

## What

`--retries N` (0 by default, at most 9) runs a failed attempt again
after the other units, within `-j`; its last try counts. `--attempts K`
is the new name of `--repeat K`, which still works.

- Retried: a rate limit, after the provider's pause; a provider, network
or setup error; an agent crash or setup timeout; a run or check out of
time while the provider refused or failed the calls.
- Not retried: a wrong key or spent credit (three in a row stop the
run), a setup this machine lacks, a missing secret, a spending or call
cap, a Stop, a failed task. A unit's cap counts all its tries.
- Before a run: a model missing from the provider's model list is
refused. A capped run fetches the price list first; offline, with a
model it cannot price, the run goes on uncapped and says so.
- Earlier tries are kept under `retried/<trial>.try-N/`: Try tabs on the
attempt's trial page, a row each in All attempts, a Retries column on
the run page. A judgement stays with its try; the run's cost counts
every try.
- The builder offers Retries per attempt on both runners; a new Harbor
recipe retries nothing unless asked (3 before).

Stacked on #146.

## Touches

the standard · backend · templates and static · docs and site

## Before and after

| Before | After |
| --- | --- |
| <img width="440" alt="Standard recipe, Limits: Attempts per problem
only"
src="https://github.com/user-attachments/assets/bbd67d22-81a7-4261-96fe-bb14674ae0a4"
/> | <img width="440" alt="Standard recipe, Limits: Attempts per problem
1 and Retries per attempt 0"
src="https://github.com/user-attachments/assets/b7eb83cd-61dc-48be-890f-b895fcc5451a"
/> |
| <img width="440" alt="Harbor recipe, Limits: Retries per trial 3"
src="https://github.com/user-attachments/assets/a5f6d0aa-bb07-4954-9d6f-52eef131e1c0"
/> | <img width="440" alt="Harbor recipe, Limits: Retries per attempt 0"
src="https://github.com/user-attachments/assets/7a38f659-e576-4ae2-bca8-b40cb00c2b58"
/> |

<img width="760" alt="Live board while attempt 2 is retried: retried
once, after infra error"
src="https://github.com/user-attachments/assets/afb56f15-4d59-46f6-9d65-39658693be9f"
/>

<img width="760" alt="Harbor recipe, common settings: Attempts per task
1"
src="https://github.com/user-attachments/assets/a019db44-212d-4e6d-ad26-6902378724df"
/>

## Verified

- `python3 app/tests/parallel.py`: 2229 tests, OK. Route suite: 547
tests, OK. The tests added with each fix fail on the code before it.
- Stand-in provider: a 429 with `Retry-After: 75` waited 75 s, then
retried; a Stop during that wait or during the retry keeps the earlier
try; a model the provider refuses (400) stops the run after three
refusals, with no retry.
- OpenRouter: a 6-unit run passed 6 of 6, its bill matching the relay's
record; a made-up model name refused in 2 s, before any call; a capped
run fetched the price list and applied the cap; with HTTPS cut off, it
ran uncapped and said so.
- Site build: every link and anchor resolves. The docs-drift check's one
finding is on `main` too.

## Risk

- Nothing is retried by default: a rate-limited unit used to run once
more (#90) and now needs `--retries 1`. A new Harbor recipe retries
nothing (3 before); a saved recipe keeps its value.
- A run now asks the provider for its model list before it starts, and a
capped run fetches the price list first: a slow list delays the start;
one that cannot be read checks nothing.
- `--repeat` keeps working; recipes now write `--attempts`.
- Undo: revert the squash commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant