Skip to content

feat(standard): force each phase's time limit, bound an agent's whole work on a unit and cap its model calls - #191

Merged
aitianshi merged 4 commits into
mainfrom
feat/step-limit-applicability
Nov 17, 2026
Merged

aitianshi merged 4 commits into
mainfrom
feat/step-limit-applicability

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #190

What

A standard run can now force its limits, and the agent's time limit covers its whole work on a unit.

  • Time limits by phase. run --timeout PHASE=SECONDS forces the limit of a phase: setup, image, agent, verify or score. Empty keeps the default, which comes from the task, then the benchmark, then Nullius. config.json records what was forced under standard.timeouts.
  • One agent limit per unit. The agent's limit is one deadline for its whole work on a unit: every turn of a conversation, and the simulated user's time between them. Before, it applied to each reply. A capability keeps one limit per add and per recall, because a memory adds a unit's sessions one by one. τ³-bench's limits.turn_seconds goes from 600 s a reply to 7200 s a conversation.
  • Turns. run --max-turns N sets the turns of a conversation. The builder offers it only on a benchmark that holds conversations.
  • A cap on model calls. run --max-calls N caps the model calls an agent makes on a unit. The relay counts each answered call on the agent's route; rate-limited and failed calls are not counted, and neither are the judge's or the simulator's. Past the cap the relay refuses the call with a 402, and the unit ends as UnitCallCapError, an agent error that is not scored. It is for agents only: a capability's route also carries the benchmark's reader, so a capability run refuses it.
  • A benchmark's own bounds. They are now options marked limit. τ³-bench gets Steps per conversation (--max-steps, from 1 up to its default of 200) and Tool errors per conversation (--max-errors). OSWorld gets Steps per task (each agent script's --max_steps); left empty it uses the chosen agent's own value, declared with the new by key.
  • Builder, Limits step. It shows Scope, then Time limits in minutes, then a section named after the benchmark when it has bounds of its own. An empty field shows the value the run uses: in the text colour when the benchmark or the agent declares it, grey when it is Nullius's default (priority: Nullius < agent < benchmark). Nothing is typed into a field for the user.

Touches

  • app/nullius/standard/: run.py, relay.py, container.py, job.py, cli.py, recipe.py, definitions.py
  • app/nullius/runs/classify.py: UnitCallCapError is an agent error
  • Catalogue: τ³-bench and OSWorld's standard.json, their drivers and upstream-settings.json
  • app/static/builder/: the Limits step
  • Docs: docs/cli/run.md, docs/standard.md, docs/builder.md, the field guide and the generated supported-benchmark pages

Before and after

The Limits step for τ³-bench, before and after:

Before After
Limits step on main: Scope only, with tasks, concurrency, attempts and two spending caps Limits step on this branch: Scope with turns and model calls, Time limits in minutes, then a tau3-bench section with steps and tool errors

Verified

  • Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging main, with feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124's serving providers and feat(standard): run each unit's agent in a Boat VM of its own #150's Boat VMs).
  • New tests:
    • A whole conversation that ends past the agent limit, though no single turn does.
    • Four one-call turns cut on the fourth call by max_calls=3.
    • The relay counts only answered calls on the agent's route.
    • A slow capability whose three adds each fit a 1 s limit while their total does not.
    • A capability run refuses --max-calls.
    • Recipe validation of the limits, including a value that rounds to 0 s.
    • τ³-bench steps above 200 are refused.
    • OSWorld's --max_steps is replaced, or added when the agent's command doesn't set it.
  • Against the installed τ³-bench upstream: the driver uses 200 steps and 10 errors by default, and 7 and 3 when the run sets them.
  • Checked in the browser, light and dark, for τ³-bench and OSWorld.

Risk

… work on a unit and cap its model calls

A run can force every phase's time limit (--timeout PHASE=SECONDS), the
turns of a conversation (--max-turns) and the model calls an agent makes
on a unit (--max-calls, refused by the relay as the agent's error). The
agent's limit now covers its whole work on a unit, every turn and the
simulated user's between them; a capability keeps a limit per add and
recall. tau3-bench's conversation limit is 7200 s.

A benchmark's own bounds are options marked `limit`: tau3-bench's steps
(1 to 200) and tool errors per conversation, OSWorld's steps per task,
each agent's own by default (`by`). The builder's Limits step shows
Scope, Time limits in minutes, then a section named after the benchmark;
choosing Original presets its agent limit.

Closes #190
…cability

# Conflicts:
#	app/nullius/standard/cli.py
#	app/nullius/standard/container.py
#	app/nullius/standard/job.py
#	app/nullius/standard/recipe.py
#	app/nullius/standard/relay.py
#	app/nullius/standard/run.py
#	app/static/builder/command-builder-core.js
#	docs/builder.md
#	docs/cli/run.md
#	docs/standard.md
@github-actions

This comment has been minimized.

…cability

# Conflicts:
#	app/nullius/standard/cli.py
#	app/nullius/standard/recipe.py
#	app/nullius/standard/run.py
#	docs/builder.md
#	docs/cli/run.md
@github-actions

This comment has been minimized.

…s value, Nullius's default greyed

An empty limit field shows the value the run uses. The benchmark's own
(its limits, a setup's own as its Original declares it, each task's) and
the agent's own read in the text colour, as a value would; Nullius's
default stays a grey placeholder. Priority, low to high: Nullius, agent,
benchmark. Choosing Original no longer types its agent limit into the
field, which forced it on every participant over each task's own.
@github-actions

Copy link
Copy Markdown

🔴 PR judge · 7 of 8 yes · #191 ↔ #190

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/runs/classify.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/standard/cli.py
Question Verdict
Changes do what the issue asks · 4 of 9 criteria ❌ no
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ✅ yes
Before and after shown · 2 images · the diff touches what is visible ✅ yes
Acceptance criteria, 4 of 9 met
# Criterion Verdict
1 run --timeout PHASE=SECONDS (setup, image, agent, verify, score) forces that phase's limit, and an unknown phase or a… ✅ met (cut)
2 The agent's limit is one deadline over the whole unit for an agent, while a capability keeps one limit per add and per … ❌ unmet (cut)
3 run --max-turns N sets a conversation's turns, and the builder offers it only when the benchmark needs `agent.convers… ❔ undecided (cut)
4 run --max-calls N makes the relay refuse the agent's call past N with a 402. The unit ends as UnitCallCapError, cla… ✅ met (cut)
5 config.json records the forced time limits, turns and calls under standard ✅ met (cut)
6 τ³-bench offers Steps per conversation (1 to 200) and Tool errors per conversation, and its driver passes them to the u… ❔ undecided (cut)
7 OSWorld offers Steps per task, which shows the chosen agent's own steps when empty and replaces or adds --max_steps w… ❌ unmet (cut)
8 The builder's Limits step shows Scope, then Time limits in minutes, then a section named after the benchmark when it ha… ✅ met (cut)
9 docs/cli/run.md, docs/standard.md, docs/builder.md and the field guide describe them ❌ unmet (cut)
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@aitianshi
aitianshi merged commit 9b7c820 into main Nov 17, 2026
2 checks passed
@aitianshi
aitianshi deleted the feat/step-limit-applicability branch November 17, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: a run cannot force its time limits, bound an agent's whole work on a unit, or cap its steps

1 participant