Repository navigation
Description
Priority
extreme. The standard's unit loop and the trial it writes (app/nullius/standard/run.py, job.py).
Components
- app/nullius (backend)
- app/web (routes)
- templates and static files
- docs and the site
- catalogue (benchmarks, agents, capabilities)
- the standard (ports, kits, relay, runs)
- CI, workflows and skills
Evidence
A standard run of an agent over ACP on main (c4c082a): nullius standard run agent-task-v1 reference --limit 1 --unsandboxed, against a local stand-in endpoint that answers in 2 s. The trial's result.json:
environment_setup 0.049 s
agent_setup (none)
agent_execution 2.108 s
verifier 0.001 s
The agent's process start, initialize and session/new (which also starts the benchmark's MCP servers) run inside agent_execution: Run.unit stamps the agent's start before drive_agent calls acp.Client.start() and new_session(). So:
- the trial page reads "Setup —", and its Agent time carries the start;
- an agent that never answers its start is written as an agent run of the whole wait, with no setup;
- the live feed's
trial.endsends one duration,agent_run, from the same stamp, and noagent_setupphase.
Screenshot
Reproduction
nullius standard run agent-task-v1 reference --model <any> --limit 1 --unsandboxed- Open the trial: the bar has no Setup segment and its chip reads "—".
- Read the trial's
result.json: noagent_setup, andagent_executionstarts whereenvironment_setupends.
Expected
A trial of an agent over ACP records its setup (its start and its session) apart from its run, as Harbor's agent_setup and agent_execution are. A start that fails is setup, not a run. The live board is told the same phases and seconds as the trial's result.
Acceptance
- A standard trial of an agent driven over ACP records
agent_setup, from the end of its environment's setup to its session opening;agent_executionand the trace's window start there. - A start that ends with no session (the agent never answered, exited, or the run was stopped) is recorded as setup, with no
agent_execution. - A command agent, a capability, an Original setup and a control record no setup.
- A task's internet door opens before the agent's phases and closes after them: neither is in
agent_setuporagent_execution. - The live feed emits an
agent_setupphase for such an agent, and itstrial.enddurations are the trial result's windows. - The field guide's phase rows hold for every runner, not Harbor alone.
Dependencies and related
The trial page's rework (#179, open; pull request #173) draws these phases and words their hovers. This is pull request #174. #173 merges first and words this phase's hover; #174 merges after it.
Activity