Repository navigation
Conversation
A unit's agent started and opened its session inside agent_execution, so the trial page had no Setup phase and Agent time carried the handshake and the benchmark's MCP servers starting. The run now notes when the session opened (an ACP agent) or the process started (a capability) and writes it as Harbor's agent_setup; agent_execution and the trace's window start there. A command agent and an upstream setup start and run in one call, so they have none.
This comment has been minimized.
This comment has been minimized.
… and tell the board the same Review of the first commit found the trial's result and the live board disagreeing, and three windows that were not what they said: - a start that ended with no session (never answered, exited, stopped) was written as the agent's run; it is now its setup, with no run, as Harbor writes a failed agent setup; - a capability got a setup of a millisecond, the time to spawn its process, which answers nothing until its first request; it has none now; - the task's internet door opened inside the agent's setup and closed inside its run; the agent now begins once it is open and ends before it closes; - the live feed still sent one duration, timed from before the setup, and no agent_setup phase; it now sends the phase, starts the turn's limit when the session opens, and takes every duration from the result. The trace-window test could not fail (its events began before the setup); it now pins the window, and the run's side has tests of its own: the four windows on shared stamps, a failed start, a capability, a conversation started again, the door. The field guide says which runs have a setup and what Other is on a standard run, and the trial page's hover no longer calls a standard agent's setup an installation.
This was referenced Nov 17, 2026
This comment has been minimized.
This comment has been minimized.
aitianshi
added a commit
that referenced
this pull request
Nov 17, 2026
… work on a unit and cap its model calls (#191) Closes #190 ## What A standard run can now force its limits, and the agent's time limit covers its whole work on a unit. - **Time limits by phase.** `run --timeout PHASE=SECONDS` forces the limit of a phase: `setup`, `image`, `agent`, `verify` or `score`. Empty keeps the default, which comes from the task, then the benchmark, then Nullius. `config.json` records what was forced under `standard.timeouts`. - **One agent limit per unit.** The agent's limit is one deadline for its whole work on a unit: every turn of a conversation, and the simulated user's time between them. Before, it applied to each reply. A capability keeps one limit per add and per recall, because a memory adds a unit's sessions one by one. τ³-bench's `limits.turn_seconds` goes from 600 s a reply to 7200 s a conversation. - **Turns.** `run --max-turns N` sets the turns of a conversation. The builder offers it only on a benchmark that holds conversations. - **A cap on model calls.** `run --max-calls N` caps the model calls an agent makes on a unit. The relay counts each answered call on the agent's route; rate-limited and failed calls are not counted, and neither are the judge's or the simulator's. Past the cap the relay refuses the call with a 402, and the unit ends as `UnitCallCapError`, an agent error that is not scored. It is for agents only: a capability's route also carries the benchmark's reader, so a capability run refuses it. - **A benchmark's own bounds.** They are now options marked `limit`. τ³-bench gets Steps per conversation (`--max-steps`, from 1 up to its default of 200) and Tool errors per conversation (`--max-errors`). OSWorld gets Steps per task (each agent script's `--max_steps`); left empty it uses the chosen agent's own value, declared with the new `by` key. - **Builder, Limits step.** It shows Scope, then Time limits in minutes, then a section named after the benchmark when it has bounds of its own. An empty field shows the value the run uses: in the text colour when the benchmark or the agent declares it, grey when it is Nullius's default (priority: Nullius < agent < benchmark). Nothing is typed into a field for the user. ## Touches - `app/nullius/standard/`: `run.py`, `relay.py`, `container.py`, `job.py`, `cli.py`, `recipe.py`, `definitions.py` - `app/nullius/runs/classify.py`: `UnitCallCapError` is an agent error - Catalogue: τ³-bench and OSWorld's `standard.json`, their drivers and `upstream-settings.json` - `app/static/builder/`: the Limits step - Docs: `docs/cli/run.md`, `docs/standard.md`, `docs/builder.md`, the field guide and the generated supported-benchmark pages ## Before and after The Limits step for τ³-bench, before and after: | Before | After | | --- | --- | | <img width="440" alt="Limits step on main: Scope only, with tasks, concurrency, attempts and two spending caps" src="https://github.com/user-attachments/assets/80b468a4-326f-4057-a4b2-5551ea93e642" /> | <img width="440" alt="Limits step on this branch: Scope with turns and model calls, Time limits in minutes, then a tau3-bench section with steps and tool errors" src="https://github.com/user-attachments/assets/6278494c-6c9f-456f-b739-9578494315ee" /> | ## Verified - Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging `main`, with #124's serving providers and #150's Boat VMs). - New tests: - A whole conversation that ends past the agent limit, though no single turn does. - Four one-call turns cut on the fourth call by `max_calls=3`. - The relay counts only answered calls on the agent's route. - A slow capability whose three adds each fit a 1 s limit while their total does not. - A capability run refuses `--max-calls`. - Recipe validation of the limits, including a value that rounds to 0 s. - τ³-bench steps above 200 are refused. - OSWorld's `--max_steps` is replaced, or added when the agent's command doesn't set it. - Against the installed τ³-bench upstream: the driver uses 200 steps and 10 errors by default, and 7 and 3 when the run sets them. - Checked in the browser, light and dark, for τ³-bench and OSWorld. ## Risk - **Agent timeouts on conversation benchmarks.** A conversation that used to fit 600 s per reply now has a single limit for the whole conversation, 7200 s on τ³-bench. A very long conversation can now time out where it didn't before. Undo by forcing a larger `--timeout agent=SECONDS`, or by reverting `limits.turn_seconds`. - **Comparability.** A run whose limits differ from the defaults is not yet marked as not comparable with the leaderboard. - **Overlap.** #174 (the agent's setup timed apart from its run) and #133 (a benchmark's own limits) change the same parts of `run.py` and the catalogue, so expect merge conflicts.
Ten commits landed on main since this branch left it; the unit loop and the trial writer conflicted. - standard/run.py, the unit loop: main runs the agent inside the unit's VM (unit_vm) and names the limits anew (agent_limit, phase_limit); this branch opens the task's door before the agent's phases and closes it after them. The VM, then the door, then the agent's setup and run, on main's limits. - standard/run.py, drive_agent: main dropped turn_limit for agent_limit, and starts the agent's clock once its session is open. The board's agent_run phase is emitted there with that same limit; the line this branch added still called turn_limit, which git merged without a conflict. - standard/run.py, unit(): over_acp() stays, with main's retry number. - standard/job.py, write_trial: both sides added parameters; all stay.
…ere it starts main lets a run force the agent's limit (--timeout agent). The board is told that limit for the agent's run, from its open session, and ACP's two start limits for its setup; the run command's page says an ACP agent starts before its limit does.
This comment has been minimized.
This comment has been minimized.
#173 merges first and words each phase's hover by the kind of run, a standard run's setup included; this branch's own wording of the same line would only conflict with it.
This comment has been minimized.
This comment has been minimized.
🟡 PR judge · 8 of 8 yes · #174 ↔ #180
Acceptance criteria, 6 of 6 met
Checks the script runs itself, 4 of 4 pass
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #180
What
A standard trial of an agent driven over ACP had no agent setup: the agent's process start,
initializeandsession/new(which also starts the benchmark's MCP servers) ran inside the window written asagent_execution. The trial page read "Setup —" and its Agent time carried the start.The run now notes when the agent's session opens and writes Harbor's
agent_setup, from the end of the environment's setup to the open session.agent_execution, and the window of the trial's trace, start there.The second commit corrects what an independent review of the first found:
agent_execution, as Harbor writes a failed agent setup.agent_setupphase, so the live board disagreed with the trial page. It emits the phase, starts the agent's limit when the session opens, and takes every duration from the trial's result.The last two commits take
main(ten commits: a unit's VM, forced time limits, attempts and retries among them):mainrenamed the turn's limit to the agent's whole work on a unit (agent_limit) and starts its clock once the session is open. The board'sagent_runphase is emitted there with that same limit, a run's--timeout agentincluded; the setup keeps ACP's two start limits. The line this branch had added still called the old name and merged without a conflict: every ACP unit would have ended in an error.agentlimit does.Installing the agent stays a stage of the run, not of a trial.
Touches
standard/run.py(the unit loop,drive_agent),standard/job.py(write_trial,feed_durations),standard/acp.py(its two start limits, named)field-guide.md(which runs have a setup; what Other is on a standard run),cli/run.md(where the agent's limit starts)test_standard_trace.py,test_standard_run.pyBefore and after
The same recipe (
agent-task-v1, the reference agent, a stand-in endpoint answering in 2 s) run on #173, which merges first, and on #173 with this branch: the trial page this lands on. Before, the agent's start sits in Agent; after, it is Setup.Verified
main(49d32d1): backend suite 2236 OK, route suite 547 OK. With feat(traces): rework the trial page around what a trial recorded #173 merged first: 2236 and 562 OK.write_trial. The trace's window is pinned as well: it starts at the open session, and a model call made during the setup widens it and is never clamped.--timeout agentand reads the feed: the setup carries ACP's 300 s, the run the forced limit. With the old limit's name left in, the setup test ends in an error.nullius standard run agent-task-v1 reference --limit 1 --unsandboxed --timeout agent=45, stand-in endpoint):result.jsonholdsenvironment_setup0.046 s,agent_setup0.054 s,agent_execution2.056 s,verifier0.001 s, back to back;events.jsonlholds the phasesenv_setup(600 s),agent_setup(300 s),agent_run(45 s),verify(600 s) and atrial.endwith the same four durations; the trace's session is 2.056 s.Risk
agent_execution. Readers already meet a trial without one (a unit that fails in its environment's setup).