Skip to content

fix(standard): time the agent's setup apart from its run - #174

Open
aitianshi wants to merge 6 commits into
mainfrom
fix/standard-agent-setup-phase
Open

aitianshi wants to merge 6 commits into
mainfrom
fix/standard-agent-setup-phase

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #180

What

A standard trial of an agent driven over ACP had no agent setup: the agent's process start, initialize and session/new (which also starts the benchmark's MCP servers) ran inside the window written as agent_execution. The trial page read "Setup —" and its Agent time carried the start.

The run now notes when the agent's session opens and writes Harbor's agent_setup, from the end of the environment's setup to the open session. agent_execution, and the window of the trial's trace, start there.

The second commit corrects what an independent review of the first found:

  • A start that ends with no session (the agent never answered, exited, or the run was stopped) was still written as the agent's run. It is its setup, with no agent_execution, as Harbor writes a failed agent setup.
  • A capability got a setup of a millisecond: the time to spawn a process that answers nothing until its first request. A capability, a command agent, an Original setup and a control have no setup phase.
  • A task's internet door opened inside the agent's setup and closed inside its run. The agent's phases begin once it is open and end before it closes.
  • The live feed still sent one duration, timed from before the setup, and no agent_setup phase, so the live board disagreed with the trial page. It emits the phase, starts the agent's limit when the session opens, and takes every duration from the trial's result.
  • The trace-window test could not fail. It pins the window now, and the run's side has tests of its own.

The last two commits take main (ten commits: a unit's VM, forced time limits, attempts and retries among them):

  • The unit loop reads: the unit's VM, the task's door, then the agent's setup and its run.
  • main renamed the turn's limit to the agent's whole work on a unit (agent_limit) and starts its clock once the session is open. The board's agent_run phase is emitted there with that same limit, a run's --timeout agent included; the setup keeps ACP's two start limits. The line this branch had added still called the old name and merged without a conflict: every ACP unit would have ended in an error.
  • The run command's page says an ACP agent starts before its agent limit does.
  • The trial page's Setup hover is left to feat(traces): rework the trial page around what a trial recorded #173, which merges first and words it for a standard run.

Installing the agent stays a stage of the run, not of a trial.

Touches

  • the standard: standard/run.py (the unit loop, drive_agent), standard/job.py (write_trial, feed_durations), standard/acp.py (its two start limits, named)
  • docs: field-guide.md (which runs have a setup; what Other is on a standard run), cli/run.md (where the agent's limit starts)
  • tests: test_standard_trace.py, test_standard_run.py

Before and after

The same recipe (agent-task-v1, the reference agent, a stand-in endpoint answering in 2 s) run on #173, which merges first, and on #173 with this branch: the trial page this lands on. Before, the agent's start sits in Agent; after, it is Setup.

Before After
Before: no Setup phase After: Setup 0.06s apart from Agent

Verified

  • On the merge with main (49d32d1): backend suite 2236 OK, route suite 547 OK. With feat(traces): rework the trial page around what a trial recorded #173 merged first: 2236 and 562 OK.
  • Seven of the new or extended tests fail on the first commit and pass on the second: the four windows on shared stamps with the feed told the same seconds; a start that never answers; a capability; a conversation the upstream starts again; the task's door (a stand-in door that takes 0.4 s to open and to close); a failed start and a trial with no clock in write_trial. The trace's window is pinned as well: it starts at the open session, and a model call made during the setup widens it and is never clamped.
  • After the merge, a new test forces --timeout agent and reads the feed: the setup carries ACP's 300 s, the run the forced limit. With the old limit's name left in, the setup test ends in an error.
  • A real local run on the merged code (nullius standard run agent-task-v1 reference --limit 1 --unsandboxed --timeout agent=45, stand-in endpoint): result.json holds environment_setup 0.046 s, agent_setup 0.054 s, agent_execution 2.056 s, verifier 0.001 s, back to back; events.jsonl holds the phases env_setup (600 s), agent_setup (300 s), agent_run (45 s), verify (600 s) and a trial.end with the same four durations; the trace's session is 2.056 s.
  • Not verified: the docker tier and the Boat backend were not run, so a real container's internet door and a unit's VM were not timed; the door's order is held by the test with the stand-in door, and the VM wraps the same block.

Risk

  • Agent time on standard runs written before and after this differs by the agent's start; a comparison across that line reads slightly different. Stored results are not rewritten, and the field guide says so.
  • On a task with an internet door, the environment's time grows by the door opening and the agent's run shrinks by its opening and closing.
  • A failed start has no agent_execution. Readers already meet a trial without one (a unit that fails in its environment's setup).
  • The live board shows "agent setup" while the agent starts, with a limit of 300 s (ACP's two start requests), then the agent's limit from its open session.
  • Merges after feat(traces): rework the trial page around what a trial recorded #173. Merged alone, the Setup hover on a standard trial would still read "installing the agent" until feat(traces): rework the trial page around what a trial recorded #173 lands.
  • Undone by a revert.

A unit's agent started and opened its session inside agent_execution, so
the trial page had no Setup phase and Agent time carried the handshake
and the benchmark's MCP servers starting. The run now notes when the
session opened (an ACP agent) or the process started (a capability) and
writes it as Harbor's agent_setup; agent_execution and the trace's window
start there. A command agent and an upstream setup start and run in one
call, so they have none.
@github-actions

This comment has been minimized.

… and tell the board the same

Review of the first commit found the trial's result and the live board
disagreeing, and three windows that were not what they said:

- a start that ended with no session (never answered, exited, stopped)
  was written as the agent's run; it is now its setup, with no run, as
  Harbor writes a failed agent setup;
- a capability got a setup of a millisecond, the time to spawn its
  process, which answers nothing until its first request; it has none now;
- the task's internet door opened inside the agent's setup and closed
  inside its run; the agent now begins once it is open and ends before it
  closes;
- the live feed still sent one duration, timed from before the setup, and
  no agent_setup phase; it now sends the phase, starts the turn's limit
  when the session opens, and takes every duration from the result.

The trace-window test could not fail (its events began before the setup);
it now pins the window, and the run's side has tests of its own: the four
windows on shared stamps, a failed start, a capability, a conversation
started again, the door. The field guide says which runs have a setup and
what Other is on a standard run, and the trial page's hover no longer
calls a standard agent's setup an installation.
@github-actions

This comment has been minimized.

aitianshi added a commit that referenced this pull request Nov 17, 2026
… work on a unit and cap its model calls (#191)

Closes #190

## What

A standard run can now force its limits, and the agent's time limit
covers its whole work on a unit.

- **Time limits by phase.** `run --timeout PHASE=SECONDS` forces the
limit of a phase: `setup`, `image`, `agent`, `verify` or `score`. Empty
keeps the default, which comes from the task, then the benchmark, then
Nullius. `config.json` records what was forced under
`standard.timeouts`.
- **One agent limit per unit.** The agent's limit is one deadline for
its whole work on a unit: every turn of a conversation, and the
simulated user's time between them. Before, it applied to each reply. A
capability keeps one limit per add and per recall, because a memory adds
a unit's sessions one by one. τ³-bench's `limits.turn_seconds` goes from
600 s a reply to 7200 s a conversation.
- **Turns.** `run --max-turns N` sets the turns of a conversation. The
builder offers it only on a benchmark that holds conversations.
- **A cap on model calls.** `run --max-calls N` caps the model calls an
agent makes on a unit. The relay counts each answered call on the
agent's route; rate-limited and failed calls are not counted, and
neither are the judge's or the simulator's. Past the cap the relay
refuses the call with a 402, and the unit ends as `UnitCallCapError`, an
agent error that is not scored. It is for agents only: a capability's
route also carries the benchmark's reader, so a capability run refuses
it.
- **A benchmark's own bounds.** They are now options marked `limit`.
τ³-bench gets Steps per conversation (`--max-steps`, from 1 up to its
default of 200) and Tool errors per conversation (`--max-errors`).
OSWorld gets Steps per task (each agent script's `--max_steps`); left
empty it uses the chosen agent's own value, declared with the new `by`
key.
- **Builder, Limits step.** It shows Scope, then Time limits in minutes,
then a section named after the benchmark when it has bounds of its own.
An empty field shows the value the run uses: in the text colour when the
benchmark or the agent declares it, grey when it is Nullius's default
(priority: Nullius < agent < benchmark). Nothing is typed into a field
for the user.

## Touches

- `app/nullius/standard/`: `run.py`, `relay.py`, `container.py`,
`job.py`, `cli.py`, `recipe.py`, `definitions.py`
- `app/nullius/runs/classify.py`: `UnitCallCapError` is an agent error
- Catalogue: τ³-bench and OSWorld's `standard.json`, their drivers and
`upstream-settings.json`
- `app/static/builder/`: the Limits step
- Docs: `docs/cli/run.md`, `docs/standard.md`, `docs/builder.md`, the
field guide and the generated supported-benchmark pages

## Before and after

The Limits step for τ³-bench, before and after:

| Before | After |
| --- | --- |
| <img width="440" alt="Limits step on main: Scope only, with tasks,
concurrency, attempts and two spending caps"
src="https://github.com/user-attachments/assets/80b468a4-326f-4057-a4b2-5551ea93e642"
/> | <img width="440" alt="Limits step on this branch: Scope with turns
and model calls, Time limits in minutes, then a tau3-bench section with
steps and tool errors"
src="https://github.com/user-attachments/assets/6278494c-6c9f-456f-b739-9578494315ee"
/> |

## Verified

- Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging
`main`, with #124's serving providers and #150's Boat VMs).
- New tests:
- A whole conversation that ends past the agent limit, though no single
turn does.
  - Four one-call turns cut on the fourth call by `max_calls=3`.
  - The relay counts only answered calls on the agent's route.
- A slow capability whose three adds each fit a 1 s limit while their
total does not.
  - A capability run refuses `--max-calls`.
- Recipe validation of the limits, including a value that rounds to 0 s.
  - τ³-bench steps above 200 are refused.
- OSWorld's `--max_steps` is replaced, or added when the agent's command
doesn't set it.
- Against the installed τ³-bench upstream: the driver uses 200 steps and
10 errors by default, and 7 and 3 when the run sets them.
- Checked in the browser, light and dark, for τ³-bench and OSWorld.

## Risk

- **Agent timeouts on conversation benchmarks.** A conversation that
used to fit 600 s per reply now has a single limit for the whole
conversation, 7200 s on τ³-bench. A very long conversation can now time
out where it didn't before. Undo by forcing a larger `--timeout
agent=SECONDS`, or by reverting `limits.turn_seconds`.
- **Comparability.** A run whose limits differ from the defaults is not
yet marked as not comparable with the leaderboard.
- **Overlap.** #174 (the agent's setup timed apart from its run) and
#133 (a benchmark's own limits) change the same parts of `run.py` and
the catalogue, so expect merge conflicts.
Ten commits landed on main since this branch left it; the unit loop and the
trial writer conflicted.

- standard/run.py, the unit loop: main runs the agent inside the unit's VM
  (unit_vm) and names the limits anew (agent_limit, phase_limit); this branch
  opens the task's door before the agent's phases and closes it after them.
  The VM, then the door, then the agent's setup and run, on main's limits.
- standard/run.py, drive_agent: main dropped turn_limit for agent_limit, and
  starts the agent's clock once its session is open. The board's agent_run
  phase is emitted there with that same limit; the line this branch added
  still called turn_limit, which git merged without a conflict.
- standard/run.py, unit(): over_acp() stays, with main's retry number.
- standard/job.py, write_trial: both sides added parameters; all stay.
…ere it starts

main lets a run force the agent's limit (--timeout agent). The board is told that limit for the agent's run, from its open session, and ACP's two start limits for its setup; the run command's page says an ACP agent starts before its limit does.
@github-actions

This comment has been minimized.

#173 merges first and words each phase's hover by the kind of run, a standard run's setup included; this branch's own wording of the same line would only conflict with it.
@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown

🟡 PR judge · 8 of 8 yes · #174 ↔ #180

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/standard/acp.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/standard/acp.py
Question Verdict
Changes do what the issue asks · 6 of 6 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ✅ yes
Before and after shown · 2 images · the diff touches what is visible ✅ yes
Acceptance criteria, 6 of 6 met
# Criterion Verdict
1 A standard trial of an agent driven over ACP records agent_setup, from the end of its environment's setup to its sess… ✅ met
2 A start that ends with no session (the agent never answered, exited, or the run was stopped) is recorded as setup, with… ✅ met
3 A command agent, a capability, an Original setup and a control record no setup. ✅ met
4 A task's internet door opens before the agent's phases and closes after them: neither is in agent_setup or `agent_exe… ✅ met
5 The live feed emits an agent_setup phase for such an agent, and its trial.end durations are the trial result's wind… ✅ met
6 The field guide says which runs have a setup, and what Other is on a standard run. ✅ met
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: a trial records no agent setup, and its agent time carries the agent's start

1 participant