Repository navigation
Description
Priority
extreme
Components
- app/nullius (backend)
- app/web (routes)
- templates and static files
- docs and the site
- catalogue (benchmarks, agents, capabilities)
- the standard (ports, kits, relay, runs)
- CI, workflows and skills
Today
An attempt is one run of a task, counted whatever its result; a retry runs an attempt again after the infrastructure or the agent failed, never because the task was failed. Harbor has both: n_attempts, 1 by default, and max_retries, 0 by default, which retries any exception but those its RetryConfig excludes (timeouts, refusals, a wrong model or key, a usage limit, the verifier's own reward errors).
The standard has --repeat K for attempts and one fixed retry: a unit that ended rate-limited runs once more after the others (#90). An agent crash, a provider error or a setup error is never retried, and once a run ends no page says an attempt was retried. The builder calls Harbor's attempts "Repeats per task" and its retries "Retries per trial", 3 by default.
Wanted
cli run --retries N, 0 by default, by Harbor's retry rule, and--attempts Kas the name of--repeat K.- The builder names both settings alike for both runners: Attempts per task (or unit) and Retries per attempt.
- A retried attempt says so on the pages.
Acceptance
-
cli run --retries N(0 to 9, default 0) runs an attempt again, up to N times, only when it ended in a retryable failure: a rate limit, a provider or network error, a setup error, an agent crash, an output-length error, a verifier error. Never an agent or verifier timeout, a refusal, a wrong model or key, a missing secret, a usage limit, the spending cap (the run's or the unit's), a stopped run, or a task the agent failed. - Retries run after the other units, one at a time and paced; nothing is retried once the run is stopped or halted, and the spending cap counts every try.
- Each earlier try is kept whole under the job's
retried/, numbered, where no reader takes it for a trial; the kept trial'sresult.jsonsays how many retries it took, andconfig.jsonrecordsretries. -
cli run --attempts Kruns each unit K times;--repeat Kstill works. - A standard recipe carries
retries; a Harbor recipe's retries default to 0; the builder labels the settings "Attempts per task" (or unit) and "Retries per attempt" for both runners; a saved recipe opens as it was saved. - A retried attempt says so on the run page, the trial page, All attempts and the live board.
-
docs/standard.md,docs/cli/run.md,docs/builder.mdand the field guide describe both settings. - Tests cover the retry rule, the rounds of retries, the kept tries, a stop, the recipe keys and the pages.
Dependencies and related
- feat(runs): read a task's repeated attempts together, by pass rate, with pass@k and pass^k #146 (the attempts read together; the field guide's "Attempts and retries"): this work is stacked on it.
- feat(standard): pace a run after a rate limit and run each rate-limited unit once more #90, the rate-limited re-run this replaces:
--retries 1gives it again. - Harbor:
n_attempts, andRetryConfig.max_retrieswith itsexclude_exceptions.
Activity
[-]Standard: no --attempts beside --repeat, and "attempt" names a repeat in the builder, recipes and driver protocol[/-][+]Standard: no --retries for an attempt that fails on the infrastructure or the agent, and --repeat is not named --attempts[/+]