Repository navigation
Labels
P0Fix now: unsafe, loses money or data, or nothing runsFix now: unsafe, loses money or data, or nothing runsenhancementNew feature or requestNew feature or request
Description
Priority
extreme
Components
- app/nullius (backend)
- app/web (routes)
- templates and static files
- docs and the site
- catalogue (benchmarks, agents, capabilities)
- the standard (ports, kits, relay, runs)
- CI, workflows and skills
Today
- A standard run's time limits cannot be set by the user. Each phase has a fixed default (setup 600 s, image 3600 s, verify per task, score 600 s), and the agent's limit comes from the task or
limits.turn_seconds. - The agent's limit applies to one reply, not to its work on the unit. On a conversation benchmark, τ³-bench's 600 s is per agent turn, so a conversation of 100 turns can take far longer than any single limit says.
- Nothing bounds the steps of an agent that loops. A model with no price leaves no spending cap either.
- A benchmark's own bounds are fixed and invisible in the builder: τ³-bench's
--max-steps 200and--max-errors 10, and each OSWorld agent's--max_steps(15, 50 or 100).
Wanted
- Every phase's time limit can be forced for every participant, from the builder and from
run. Empty keeps the default. - The agent's limit covers its whole work on a unit: every turn, and the simulated user's time between them.
- Turns per conversation can be set, and only on a benchmark that holds conversations.
- An optional cap on the model calls an agent makes on a unit, counted over the whole unit.
- A benchmark's own bounds are options a run can set, shown in their own section of Limits.
- An empty limit field shows the value the run will use: the benchmark's or the agent's own in the text colour, Nullius's default in grey (priority: Nullius < agent < benchmark).
Acceptance
-
run --timeout PHASE=SECONDS(setup, image, agent, verify, score) forces that phase's limit, and an unknown phase or a non-positive value is refused - The agent's limit is one deadline over the whole unit for an agent, while a capability keeps one limit per add and per recall
-
run --max-turns Nsets a conversation's turns, and the builder offers it only when the benchmark needsagent.conversation/v1 -
run --max-calls Nmakes the relay refuse the agent's call past N with a 402. The unit ends asUnitCallCapError, classified as an agent error and not scored. Rate-limited or failed calls, and the judge's and the simulator's calls, are not counted. A capability run refuses it -
config.jsonrecords the forced time limits, turns and calls understandard - τ³-bench offers Steps per conversation (1 to 200) and Tool errors per conversation, and its driver passes them to the upstream's run config
- OSWorld offers Steps per task, which shows the chosen agent's own steps when empty and replaces or adds
--max_stepswhen set - The builder's Limits step shows Scope, then Time limits in minutes, then a section named after the benchmark when it has bounds of its own
-
docs/cli/run.md,docs/standard.md,docs/builder.mdand the field guide describe them
Dependencies and related
- A spending cap is the run's alone: one unit can spend all of it #47 (a spending cap per unit): the call cap uses the same relay refusal path
- fix(standard): time the agent's setup apart from its run #174 (time the agent's setup apart from its run) and feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control #133 (a benchmark sets its own limits) touch the same code
Metadata
Metadata
Assignees
Labels
P0Fix now: unsafe, loses money or data, or nothing runsFix now: unsafe, loses money or data, or nothing runsenhancementNew feature or requestNew feature or request
Activity