Repository navigation
feat(standard): force each phase's time limit, bound an agent's whole work on a unit and cap its model calls - #191
Merged
Conversation
… work on a unit and cap its model calls A run can force every phase's time limit (--timeout PHASE=SECONDS), the turns of a conversation (--max-turns) and the model calls an agent makes on a unit (--max-calls, refused by the relay as the agent's error). The agent's limit now covers its whole work on a unit, every turn and the simulated user's between them; a capability keeps a limit per add and recall. tau3-bench's conversation limit is 7200 s. A benchmark's own bounds are options marked `limit`: tau3-bench's steps (1 to 200) and tool errors per conversation, OSWorld's steps per task, each agent's own by default (`by`). The builder's Limits step shows Scope, Time limits in minutes, then a section named after the benchmark; choosing Original presets its agent limit. Closes #190
…cability # Conflicts: # app/nullius/standard/cli.py # app/nullius/standard/container.py # app/nullius/standard/job.py # app/nullius/standard/recipe.py # app/nullius/standard/relay.py # app/nullius/standard/run.py # app/static/builder/command-builder-core.js # docs/builder.md # docs/cli/run.md # docs/standard.md
This comment has been minimized.
This comment has been minimized.
…cability # Conflicts: # app/nullius/standard/cli.py # app/nullius/standard/recipe.py # app/nullius/standard/run.py # docs/builder.md # docs/cli/run.md
This comment has been minimized.
This comment has been minimized.
…s value, Nullius's default greyed An empty limit field shows the value the run uses. The benchmark's own (its limits, a setup's own as its Original declares it, each task's) and the agent's own read in the text colour, as a value would; Nullius's default stays a grey placeholder. Priority, low to high: Nullius, agent, benchmark. Choosing Original no longer types its agent limit into the field, which forced it on every participant over each task's own.
🔴 PR judge · 7 of 8 yes · #191 ↔ #190
Acceptance criteria, 4 of 9 met
Checks the script runs itself, 4 of 4 pass
some of the diff was cut before the judge read it |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #190
What
A standard run can now force its limits, and the agent's time limit covers its whole work on a unit.
run --timeout PHASE=SECONDSforces the limit of a phase:setup,image,agent,verifyorscore. Empty keeps the default, which comes from the task, then the benchmark, then Nullius.config.jsonrecords what was forced understandard.timeouts.limits.turn_secondsgoes from 600 s a reply to 7200 s a conversation.run --max-turns Nsets the turns of a conversation. The builder offers it only on a benchmark that holds conversations.run --max-calls Ncaps the model calls an agent makes on a unit. The relay counts each answered call on the agent's route; rate-limited and failed calls are not counted, and neither are the judge's or the simulator's. Past the cap the relay refuses the call with a 402, and the unit ends asUnitCallCapError, an agent error that is not scored. It is for agents only: a capability's route also carries the benchmark's reader, so a capability run refuses it.limit. τ³-bench gets Steps per conversation (--max-steps, from 1 up to its default of 200) and Tool errors per conversation (--max-errors). OSWorld gets Steps per task (each agent script's--max_steps); left empty it uses the chosen agent's own value, declared with the newbykey.Touches
app/nullius/standard/:run.py,relay.py,container.py,job.py,cli.py,recipe.py,definitions.pyapp/nullius/runs/classify.py:UnitCallCapErroris an agent errorstandard.json, their drivers andupstream-settings.jsonapp/static/builder/: the Limits stepdocs/cli/run.md,docs/standard.md,docs/builder.md, the field guide and the generated supported-benchmark pagesBefore and after
The Limits step for τ³-bench, before and after:
Verified
main, with feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124's serving providers and feat(standard): run each unit's agent in a Boat VM of its own #150's Boat VMs).max_calls=3.--max-calls.--max_stepsis replaced, or added when the agent's command doesn't set it.Risk
--timeout agent=SECONDS, or by revertinglimits.turn_seconds.run.pyand the catalogue, so expect merge conflicts.