Skip to content

feat(standard): retry an attempt that fails on the infrastructure or the agent, with --retries beside --attempts - #154

Merged
aitianshi merged 20 commits into
mainfrom
feat/standard-attempts
Nov 17, 2026
Merged

aitianshi merged 20 commits into
mainfrom
feat/standard-attempts

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #152

What

--retries N (0 by default, at most 9) runs a failed attempt again after the other units, within -j; its last try counts. --attempts K is the new name of --repeat K, which still works.

  • Retried: a rate limit, after the provider's pause; a provider, network or setup error; an agent crash or setup timeout; a run or check out of time while the provider refused or failed the calls.
  • Not retried: a wrong key or spent credit (three in a row stop the run), a setup this machine lacks, a missing secret, a spending or call cap, a Stop, a failed task. A unit's cap counts all its tries.
  • Before a run: a model missing from the provider's model list is refused. A capped run fetches the price list first; offline, with a model it cannot price, the run goes on uncapped and says so.
  • Earlier tries are kept under retried/<trial>.try-N/: Try tabs on the attempt's trial page, a row each in All attempts, a Retries column on the run page. A judgement stays with its try; the run's cost counts every try.
  • The builder offers Retries per attempt on both runners; a new Harbor recipe retries nothing unless asked (3 before).

Stacked on #146.

Touches

the standard · backend · templates and static · docs and site

Before and after

Before After
Standard recipe, Limits: Attempts per problem only Standard recipe, Limits: Attempts per problem 1 and Retries per attempt 0
Harbor recipe, Limits: Retries per trial 3 Harbor recipe, Limits: Retries per attempt 0
Live board while attempt 2 is retried: retried once, after infra error Harbor recipe, common settings: Attempts per task 1

Verified

  • python3 app/tests/parallel.py: 2229 tests, OK. Route suite: 547 tests, OK. The tests added with each fix fail on the code before it.
  • Stand-in provider: a 429 with Retry-After: 75 waited 75 s, then retried; a Stop during that wait or during the retry keeps the earlier try; a model the provider refuses (400) stops the run after three refusals, with no retry.
  • OpenRouter: a 6-unit run passed 6 of 6, its bill matching the relay's record; a made-up model name refused in 2 s, before any call; a capped run fetched the price list and applied the cap; with HTTPS cut off, it ran uncapped and said so.
  • Site build: every link and anchor resolves. The docs-drift check's one finding is on main too.

Risk

  • Nothing is retried by default: a rate-limited unit used to run once more (feat(standard): pace a run after a rate limit and run each rate-limited unit once more #90) and now needs --retries 1. A new Harbor recipe retries nothing (3 before); a saved recipe keeps its value.
  • A run now asks the provider for its model list before it starts, and a capped run fetches the price list first: a slow list delays the start; one that cannot be read checks nothing.
  • --repeat keeps working; recipes now write --attempts.
  • Undo: revert the squash commit.

…ith pass@k and pass^k

Trials of one task in one job are its attempts, numbered by when they
started (nullius.runs.attempts). The trial page gets a tab per attempt and
an All attempts table; the job page shows one row per task with its pass
count, the instruction score once and the mean tokens and times; the run
page and Compare show pass@k and pass^k. Compare judges a task by its pass
rate over every attempt, ranks by how far it moved (read with one pass and
one fail added, so a single flipped attempt ranks below a steady gap), and
its side-by-side page gives each run its own attempt tabs.

Attempts are read as the adjusted rate reads trials: an infra attempt is
counted apart and left out of the pass count, the means and pass@k; an
agent's timeout or crash counts as a failure. The standard's pass_hat now
calls the same estimators; what it writes is unchanged.
…nt-sorting

# Conflicts:
#	app/static/style.css
A repeat is one run of a task, counted whatever its result (Harbor's
n_attempts, the standard's --repeat). An attempt is one try within a
repeat, taken again only when the infrastructure or the agent fails, and
only its last try is kept as the repeat's trial. The pages, Compare and its
command's JSON, the field guide and the module (nullius.runs.repeats) now
say repeat where they meant one. The standard still writes pass_hat_k's
`attempts` key, the name result.json has always carried.
…tructure or the agent fails

A repeat is one run of a unit, counted whatever its result (--repeat); an
attempt is one try within a repeat. A repeat that ends in a failure a
further try may mend (Harbor's retry rule: a rate limit, a provider,
network or setup error, an agent crash, an output-length or verifier
error) runs again after the other units, one at a time and paced, until it
ends otherwise or has had --attempts tries, 2 by default: the single re-run
of a rate-limited unit, extended to every retryable failure. Never a
timeout, a refusal, a wrong model or key, a missing secret, a usage limit,
the spending cap, a stop, or a task the agent failed. Each earlier try is
kept in retried/<trial>.attempt-N; the kept trial records its attempt and
the config records attempts.

Both runners' recipes carry repeat and attempts: a Harbor recipe is saved
as version 2 (repeat is --n-attempts, attempts is --max-retries + 1) and
version 1 is read as it was saved. Where "attempt" named a repeat it now
says repeat: the builder's labels, budgets, benchmark notes, and the
driver protocol, which sends repeat to a definition declaring
limits.repeats and attempt to one still declaring limits.attempts.
@github-actions

This comment has been minimized.

…the run does

An attempt is one run of a task, counted whatever its result (Harbor's
n_attempts, the standard's --repeat); a retry is a further try of an
attempt after an infrastructure or agent failure, and only its last try is
kept as the attempt's trial. This takes back the word repeat for the first,
as Harbor names it, and the field guide's section says both.

A task's attempts are numbered by the run's own number when every trial
carries one (the standard's ..._u0042r2), so an attempt retried after the
others keeps its place; by start time otherwise, as Harbor's names carry no
number.
…the agent, with --retries beside --attempts

Merges #146 and rebuilds this change in its words. An attempt is one run
of a unit, counted whatever its result: --attempts K, the older --repeat K
still read. A retry runs an attempt again after a failure a further try
may mend, by Harbor's retry rule (a rate limit, a provider, network or
setup error, an agent crash, an output-length or verifier error; never a
timeout, a refusal, a wrong model or key, a missing secret, a usage limit,
the spending cap, a stop, or a task the agent failed): --retries N, 0 by
default, so the single re-run of a rate-limited unit now needs --retries 1.

Earlier tries are kept in retried/<trial>.try-N, the kept trial records
its retries (standard.retries) and the config records retries. The builder
says Attempts per task and Retries per attempt for both runners, and a
Harbor recipe's retries default to 0. A retried attempt says so on the run
page, the trial page, All attempts and the live board, and the progress
counts every try. The driver protocol, the catalogue and the recipe formats
keep main's words.
@github-actions

This comment has been minimized.

@aitianshi aitianshi changed the title feat(standard): add --attempts, trying a repeat again when the infrastructure or the agent fails feat(standard): retry an attempt that fails on the infrastructure or the agent, with --retries beside --attempts Nov 16, 2026
…at a further try may mend

A retried attempt's tries are tabs of their own on its trial page (Try 1,
Try 2, the last counted), and a faded row each in All attempts; the run
page and All attempts count each attempt's retries against the run's limit.

A retry follows only a failure a second try may mend: never a wrong key or
model, spent credit, a time run out, a setup the machine cannot give, or an
attempt that spent its own cap (the cap counts all of its tries together).
A rate limit's retry waits the provider's pause, or a minute. A Stop during
a retry keeps the try before; a merge keeps the shards' tries; a host run's
copy, a judgement, the progress and the live board each read the retry as
its own try. Proof runs retry a rate limit once.
…hanges a reading

Keep main's pass@k and pass^k text; add only the words for attempt and retry, which tasks the pages leave out, how a task's pass rate counts and when a Compare row disagrees.
…empts

Brings in main through #146. The conflicts are resolved as in the rebased branch the suites passed on; the resolution also cuts the retries docs and the --retries help to the AGENTS.md documentation rule.
… and say the pause in whole seconds

A rate-limited attempt stays on the run page while its retry waits, and a
Stop during the wait leaves nothing to put back; a stopped retry's empty
retried/ folder goes.
…ks and attempts apart in its last line

A retry round shares the run's concurrency (-j) instead of running one at a
time; a retry stopped under it still leaves the try before. The run's last
line says the tasks and, when each was attempted more than once, the
attempts beside them ("3 problems, 6 attempts"), never the attempts as tasks.
Each earlier try was paid for: the job's total cost adds what each
attempt's tries before its last recorded (Trial.retried_cost_usd), and the
live board's total keeps an attempt's cost when its retry starts.
A try's page found its try in the attempt's tries, then read them all again
for its tabs.
…m's checks included

The catalogue's notes, conform's flag and its pages, the import-agent skill
and the run's help name the flag --attempts; --repeat is still accepted.
…st, and fetch the price list a cap needs

A model the provider's list lacks would be refused on every call: the run is
refused before anything installs (a :suffix routes the model it follows; a
provider with no list is not checked). A capped run fetches the model price
list first; when it cannot be fetched and a model has no price, the run goes
on without the cap and says so, in its log and on its units. No test reaches
past this machine: tests/__init__.py refuses any other address.
…ck among the run's refusals

The standard's Attempts and Retries rows link the field guide and the run page instead of restating them; the retries table keeps which failures are tried again and drops how many run at once and where earlier tries are kept.
@github-actions

This comment has been minimized.

Base automatically changed from feat/compare-pass-count-sorting to main November 17, 2026 22:19
@github-actions

Copy link
Copy Markdown

🟡 PR judge · 7 of 8 yes · #154 ↔ #152

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/execution.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/execution.py
Question Verdict
Changes do what the issue asks · 8 of 8 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ⚠️ partly
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ✅ yes
Before and after shown · 6 images · the diff touches what is visible ✅ yes
Acceptance criteria, 8 of 8 met
# Criterion Verdict
1 cli run --retries N (0 to 9, default 0) runs an attempt again, up to N times, only when it ended in a retryable failu… ✅ met (cut)
2 Retries run after the other units, one at a time and paced; nothing is retried once the run is stopped or halted, and t… ✅ met (cut)
3 Each earlier try is kept whole under the job's retried/, numbered, where no reader takes it for a trial; the kept tri… ✅ met (cut)
4 cli run --attempts K runs each unit K times; --repeat K still works. ✅ met (cut)
5 A standard recipe carries retries; a Harbor recipe's retries default to 0; the builder labels the settings "Attempts … ✅ met (cut)
6 A retried attempt says so on the run page, the trial page, All attempts and the live board. ✅ met (cut)
7 docs/standard.md, docs/cli/run.md, docs/builder.md and the field guide describe both settings. ✅ met (cut)
8 Tests cover the retry rule, the rounds of retries, the kept tries, a stop, the recipe keys and the pages. ✅ met (cut)
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@aitianshi
aitianshi merged commit 49d32d1 into main Nov 17, 2026
2 checks passed
@aitianshi
aitianshi deleted the feat/standard-attempts branch November 17, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: no --retries for an attempt that fails on the infrastructure or the agent, and --repeat is not named --attempts

1 participant