Repository navigation
feat(standard): retry an attempt that fails on the infrastructure or the agent, with --retries beside --attempts - #154
Merged
Merged
Conversation
…ith pass@k and pass^k Trials of one task in one job are its attempts, numbered by when they started (nullius.runs.attempts). The trial page gets a tab per attempt and an All attempts table; the job page shows one row per task with its pass count, the instruction score once and the mean tokens and times; the run page and Compare show pass@k and pass^k. Compare judges a task by its pass rate over every attempt, ranks by how far it moved (read with one pass and one fail added, so a single flipped attempt ranks below a steady gap), and its side-by-side page gives each run its own attempt tabs. Attempts are read as the adjusted rate reads trials: an infra attempt is counted apart and left out of the pass count, the means and pass@k; an agent's timeout or crash counts as a failure. The standard's pass_hat now calls the same estimators; what it writes is unchanged.
…nt-sorting # Conflicts: # app/static/style.css
A repeat is one run of a task, counted whatever its result (Harbor's n_attempts, the standard's --repeat). An attempt is one try within a repeat, taken again only when the infrastructure or the agent fails, and only its last try is kept as the repeat's trial. The pages, Compare and its command's JSON, the field guide and the module (nullius.runs.repeats) now say repeat where they meant one. The standard still writes pass_hat_k's `attempts` key, the name result.json has always carried.
…tructure or the agent fails A repeat is one run of a unit, counted whatever its result (--repeat); an attempt is one try within a repeat. A repeat that ends in a failure a further try may mend (Harbor's retry rule: a rate limit, a provider, network or setup error, an agent crash, an output-length or verifier error) runs again after the other units, one at a time and paced, until it ends otherwise or has had --attempts tries, 2 by default: the single re-run of a rate-limited unit, extended to every retryable failure. Never a timeout, a refusal, a wrong model or key, a missing secret, a usage limit, the spending cap, a stop, or a task the agent failed. Each earlier try is kept in retried/<trial>.attempt-N; the kept trial records its attempt and the config records attempts. Both runners' recipes carry repeat and attempts: a Harbor recipe is saved as version 2 (repeat is --n-attempts, attempts is --max-retries + 1) and version 1 is read as it was saved. Where "attempt" named a repeat it now says repeat: the builder's labels, budgets, benchmark notes, and the driver protocol, which sends repeat to a definition declaring limits.repeats and attempt to one still declaring limits.attempts.
This comment has been minimized.
This comment has been minimized.
…rd's retry mark for both runners
…the run does An attempt is one run of a task, counted whatever its result (Harbor's n_attempts, the standard's --repeat); a retry is a further try of an attempt after an infrastructure or agent failure, and only its last try is kept as the attempt's trial. This takes back the word repeat for the first, as Harbor names it, and the field guide's section says both. A task's attempts are numbered by the run's own number when every trial carries one (the standard's ..._u0042r2), so an attempt retried after the others keeps its place; by start time otherwise, as Harbor's names carry no number.
…the agent, with --retries beside --attempts Merges #146 and rebuilds this change in its words. An attempt is one run of a unit, counted whatever its result: --attempts K, the older --repeat K still read. A retry runs an attempt again after a failure a further try may mend, by Harbor's retry rule (a rate limit, a provider, network or setup error, an agent crash, an output-length or verifier error; never a timeout, a refusal, a wrong model or key, a missing secret, a usage limit, the spending cap, a stop, or a task the agent failed): --retries N, 0 by default, so the single re-run of a rate-limited unit now needs --retries 1. Earlier tries are kept in retried/<trial>.try-N, the kept trial records its retries (standard.retries) and the config records retries. The builder says Attempts per task and Retries per attempt for both runners, and a Harbor recipe's retries default to 0. A retried attempt says so on the run page, the trial page, All attempts and the live board, and the progress counts every try. The driver protocol, the catalogue and the recipe formats keep main's words.
This comment has been minimized.
This comment has been minimized.
…at a further try may mend A retried attempt's tries are tabs of their own on its trial page (Try 1, Try 2, the last counted), and a faded row each in All attempts; the run page and All attempts count each attempt's retries against the run's limit. A retry follows only a failure a second try may mend: never a wrong key or model, spent credit, a time run out, a setup the machine cannot give, or an attempt that spent its own cap (the cap counts all of its tries together). A rate limit's retry waits the provider's pause, or a minute. A Stop during a retry keeps the try before; a merge keeps the shards' tries; a host run's copy, a judgement, the progress and the live board each read the retry as its own try. Proof runs retry a rate limit once.
…hanges a reading Keep main's pass@k and pass^k text; add only the words for attempt and retry, which tasks the pages leave out, how a task's pass rate counts and when a Compare row disagrees.
…empts Brings in main through #146. The conflicts are resolved as in the rebased branch the suites passed on; the resolution also cuts the retries docs and the --retries help to the AGENTS.md documentation rule.
… and say the pause in whole seconds A rate-limited attempt stays on the run page while its retry waits, and a Stop during the wait leaves nothing to put back; a stopped retry's empty retried/ folder goes.
…ks and attempts apart in its last line
A retry round shares the run's concurrency (-j) instead of running one at a
time; a retry stopped under it still leaves the try before. The run's last
line says the tasks and, when each was attempted more than once, the
attempts beside them ("3 problems, 6 attempts"), never the attempts as tasks.
Each earlier try was paid for: the job's total cost adds what each attempt's tries before its last recorded (Trial.retried_cost_usd), and the live board's total keeps an attempt's cost when its retry starts.
A try's page found its try in the attempt's tries, then read them all again for its tabs.
…m's checks included The catalogue's notes, conform's flag and its pages, the import-agent skill and the run's help name the flag --attempts; --repeat is still accepted.
…st, and fetch the price list a cap needs A model the provider's list lacks would be refused on every call: the run is refused before anything installs (a :suffix routes the model it follows; a provider with no list is not checked). A capped run fetches the model price list first; when it cannot be fetched and a model has no price, the run goes on without the cap and says so, in its log and on its units. No test reaches past this machine: tests/__init__.py refuses any other address.
…ck among the run's refusals The standard's Attempts and Retries rows link the field guide and the run page instead of restating them; the retries table keeps which failures are tried again and drops how many run at once and where earlier tries are kept.
This comment has been minimized.
This comment has been minimized.
🟡 PR judge · 7 of 8 yes · #154 ↔ #152
Acceptance criteria, 8 of 8 met
Checks the script runs itself, 4 of 4 pass
some of the diff was cut before the judge read it |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #152
What
--retries N(0 by default, at most 9) runs a failed attempt again after the other units, within-j; its last try counts.--attempts Kis the new name of--repeat K, which still works.retried/<trial>.try-N/: Try tabs on the attempt's trial page, a row each in All attempts, a Retries column on the run page. A judgement stays with its try; the run's cost counts every try.Stacked on #146.
Touches
the standard · backend · templates and static · docs and site
Before and after
Verified
python3 app/tests/parallel.py: 2229 tests, OK. Route suite: 547 tests, OK. The tests added with each fix fail on the code before it.Retry-After: 75waited 75 s, then retried; a Stop during that wait or during the retry keeps the earlier try; a model the provider refuses (400) stops the run after three refusals, with no retry.maintoo.Risk
--retries 1. A new Harbor recipe retries nothing (3 before); a saved recipe keeps its value.--repeatkeeps working; recipes now write--attempts.