Repository navigation
feat(runs): read a task's repeated attempts together, by pass rate, with pass@k and pass^k - #146
Merged
Merged
Conversation
…ith pass@k and pass^k Trials of one task in one job are its attempts, numbered by when they started (nullius.runs.attempts). The trial page gets a tab per attempt and an All attempts table; the job page shows one row per task with its pass count, the instruction score once and the mean tokens and times; the run page and Compare show pass@k and pass^k. Compare judges a task by its pass rate over every attempt, ranks by how far it moved (read with one pass and one fail added, so a single flipped attempt ranks below a steady gap), and its side-by-side page gives each run its own attempt tabs. Attempts are read as the adjusted rate reads trials: an infra attempt is counted apart and left out of the pass count, the means and pass@k; an agent's timeout or crash counts as a failure. The standard's pass_hat now calls the same estimators; what it writes is unchanged.
…nt-sorting # Conflicts: # app/static/style.css
Closed
3 of 15 tasks
This comment has been minimized.
This comment has been minimized.
A repeat is one run of a task, counted whatever its result (Harbor's n_attempts, the standard's --repeat). An attempt is one try within a repeat, taken again only when the infrastructure or the agent fails, and only its last try is kept as the repeat's trial. The pages, Compare and its command's JSON, the field guide and the module (nullius.runs.repeats) now say repeat where they meant one. The standard still writes pass_hat_k's `attempts` key, the name result.json has always carried.
This comment has been minimized.
This comment has been minimized.
…the run does An attempt is one run of a task, counted whatever its result (Harbor's n_attempts, the standard's --repeat); a retry is a further try of an attempt after an infrastructure or agent failure, and only its last try is kept as the attempt's trial. This takes back the word repeat for the first, as Harbor names it, and the field guide's section says both. A task's attempts are numbered by the run's own number when every trial carries one (the standard's ..._u0042r2), so an attempt retried after the others keeps its place; by start time otherwise, as Harbor's names carry no number.
This comment has been minimized.
This comment has been minimized.
aitianshi
added a commit
that referenced
this pull request
Nov 16, 2026
…the agent, with --retries beside --attempts Merges #146 and rebuilds this change in its words. An attempt is one run of a unit, counted whatever its result: --attempts K, the older --repeat K still read. A retry runs an attempt again after a failure a further try may mend, by Harbor's retry rule (a rate limit, a provider, network or setup error, an agent crash, an output-length or verifier error; never a timeout, a refusal, a wrong model or key, a missing secret, a usage limit, the spending cap, a stop, or a task the agent failed): --retries N, 0 by default, so the single re-run of a rate-limited unit now needs --retries 1. Earlier tries are kept in retried/<trial>.try-N, the kept trial records its retries (standard.retries) and the config records retries. The builder says Attempts per task and Retries per attempt for both runners, and a Harbor recipe's retries default to 0. A retried attempt says so on the run page, the trial page, All attempts and the live board, and the progress counts every try. The driver protocol, the catalogue and the recipe formats keep main's words.
This was referenced Nov 16, 2026
…hanges a reading Keep main's pass@k and pass^k text; add only the words for attempt and retry, which tasks the pages leave out, how a task's pass rate counts and when a Compare row disagrees.
aitianshi
added a commit
that referenced
this pull request
Nov 17, 2026
…empts Brings in main through #146. The conflicts are resolved as in the rebased branch the suites passed on; the resolution also cuts the retries docs and the --retries help to the AGENTS.md documentation rule.
🟡 PR judge · 6 of 8 yes · #146 ↔ #130
Acceptance criteria, 8 of 8 met
Checks the script runs itself, 4 of 4 pass
some of the diff was cut before the judge read it |
aitianshi
added a commit
that referenced
this pull request
Nov 17, 2026
…the agent, with --retries beside --attempts (#154) Closes #152 ## What `--retries N` (0 by default, at most 9) runs a failed attempt again after the other units, within `-j`; its last try counts. `--attempts K` is the new name of `--repeat K`, which still works. - Retried: a rate limit, after the provider's pause; a provider, network or setup error; an agent crash or setup timeout; a run or check out of time while the provider refused or failed the calls. - Not retried: a wrong key or spent credit (three in a row stop the run), a setup this machine lacks, a missing secret, a spending or call cap, a Stop, a failed task. A unit's cap counts all its tries. - Before a run: a model missing from the provider's model list is refused. A capped run fetches the price list first; offline, with a model it cannot price, the run goes on uncapped and says so. - Earlier tries are kept under `retried/<trial>.try-N/`: Try tabs on the attempt's trial page, a row each in All attempts, a Retries column on the run page. A judgement stays with its try; the run's cost counts every try. - The builder offers Retries per attempt on both runners; a new Harbor recipe retries nothing unless asked (3 before). Stacked on #146. ## Touches the standard · backend · templates and static · docs and site ## Before and after | Before | After | | --- | --- | | <img width="440" alt="Standard recipe, Limits: Attempts per problem only" src="https://github.com/user-attachments/assets/bbd67d22-81a7-4261-96fe-bb14674ae0a4" /> | <img width="440" alt="Standard recipe, Limits: Attempts per problem 1 and Retries per attempt 0" src="https://github.com/user-attachments/assets/b7eb83cd-61dc-48be-890f-b895fcc5451a" /> | | <img width="440" alt="Harbor recipe, Limits: Retries per trial 3" src="https://github.com/user-attachments/assets/a5f6d0aa-bb07-4954-9d6f-52eef131e1c0" /> | <img width="440" alt="Harbor recipe, Limits: Retries per attempt 0" src="https://github.com/user-attachments/assets/7a38f659-e576-4ae2-bca8-b40cb00c2b58" /> | <img width="760" alt="Live board while attempt 2 is retried: retried once, after infra error" src="https://github.com/user-attachments/assets/afb56f15-4d59-46f6-9d65-39658693be9f" /> <img width="760" alt="Harbor recipe, common settings: Attempts per task 1" src="https://github.com/user-attachments/assets/a019db44-212d-4e6d-ad26-6902378724df" /> ## Verified - `python3 app/tests/parallel.py`: 2229 tests, OK. Route suite: 547 tests, OK. The tests added with each fix fail on the code before it. - Stand-in provider: a 429 with `Retry-After: 75` waited 75 s, then retried; a Stop during that wait or during the retry keeps the earlier try; a model the provider refuses (400) stops the run after three refusals, with no retry. - OpenRouter: a 6-unit run passed 6 of 6, its bill matching the relay's record; a made-up model name refused in 2 s, before any call; a capped run fetched the price list and applied the cap; with HTTPS cut off, it ran uncapped and said so. - Site build: every link and anchor resolves. The docs-drift check's one finding is on `main` too. ## Risk - Nothing is retried by default: a rate-limited unit used to run once more (#90) and now needs `--retries 1`. A new Harbor recipe retries nothing (3 before); a saved recipe keeps its value. - A run now asks the provider for its model list before it starts, and a capped run fetches the price list first: a slow list delays the start; one that cannot be read checks nothing. - `--repeat` keeps working; recipes now write `--attempts`. - Undo: revert the squash commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #130
Closes #145
What
An attempt is one run of a task, counted whatever its result (Harbor's
n_attempts, the standard's--repeat); a retry is a further try of an attempt after an infrastructure or agent failure, and only its last try is kept as the attempt's trial. A task's attempts in one job are now read together, by one backend module (nullius.runs.attempts), the way the run's adjusted rate reads trials: an infra attempt is counted apart and left out of the pass count, the means and pass@k, and an agent's timeout or crash counts as a failure. Attempts are numbered by the run's own number when its trials carry one (the standard'su0000r2), so an attempt retried after the others keeps its place, else by start time. The trial page gets a tab per attempt and an All attempts table. The job page shows one row per repeated task (its pass count, a square per attempt, the instruction score once, the mean tokens and times) that opens onto its attempts, and the run page shows pass@k and pass^k. Compare (#130) judges a task by its pass rate over every attempt instead of its last one, ranks the rows by how far that rate moved (read with one pass and one fail added, so a single flipped attempt ranks below a steady gap), shows pass@k and pass^k in its run summary, and gives each side of the side-by-side page its own attempt tabs. The standard'spass_hatcalls the same estimators; what it writes is unchanged.Touches
backend · routes · templates and static · docs and site · the standard (
pass_hatcalls the shared estimators)Before and after
An infra attempt counted apart, after (a hand-made run in Harbor's layout): t1 reads 1/2 · 1 infra and its means leave the rate-limited attempt out; t2's timeout counts as a failure.
The screenshots are two real standard runs of GSM8K against a stand-in model server (one problem, three attempts each); the instruction scores are stand-in records.
Verified
main(9b7c820):python3 app/tests/parallel.py, 2198 tests, OK;uv run --no-project --with-requirements app/requirements.txt python app/tests/parallel.py app/ui_tests, 542 tests, OK.test_attempts.py(numbering, by the run's own number or by start; grouping, infra apart, the verdict rule, token means, the pass@k rules),test_comparison.py(disagreement by pass rate and by endings, single attempts, infra, ranking, a pass rate never set against a mean score, pass@k over the rows' tasks),test_attempt_routes.pyandtest_comparison_routes.py(tabs, All attempts, grouped rows with their means and infra, pass@k on the run page, the counts in Compare and side by side).attempts.Attemptsis pinned. The docs-drift check's two findings (tau3-bench/search_relay,standard/pace) are onmaintoo.Risk
docs/field-guide.md#attemptsstates the rules, and the tests pin them.board.jsfails they stay hidden, and the trial tabs and All attempts still list them.pass_hatruns inwrite_resultat the end of every standard run, so an error in it would leave that run without itsresult.json. It now calls the shared estimators (imported inside the function);test_pass_hat_k_as_tau_bench_counts_itpins its output, which is unchanged, and the shard tests (test_standard_shard.py) write their results through it.