Skip to content

Standard: a trial records no agent setup, and its agent time carries the agent's start #180

Closed

Description

@aitianshi

Priority

extreme. The standard's unit loop and the trial it writes (app/nullius/standard/run.py, job.py).

Components

  • app/nullius (backend)
  • app/web (routes)
  • templates and static files
  • docs and the site
  • catalogue (benchmarks, agents, capabilities)
  • the standard (ports, kits, relay, runs)
  • CI, workflows and skills

Evidence

A standard run of an agent over ACP on main (c4c082a): nullius standard run agent-task-v1 reference --limit 1 --unsandboxed, against a local stand-in endpoint that answers in 2 s. The trial's result.json:

environment_setup  0.049 s
agent_setup        (none)
agent_execution    2.108 s
verifier           0.001 s

The agent's process start, initialize and session/new (which also starts the benchmark's MCP servers) run inside agent_execution: Run.unit stamps the agent's start before drive_agent calls acp.Client.start() and new_session(). So:

  • the trial page reads "Setup —", and its Agent time carries the start;
  • an agent that never answers its start is written as an agent run of the whole wait, with no setup;
  • the live feed's trial.end sends one duration, agent_run, from the same stamp, and no agent_setup phase.

Screenshot

A standard trial on main: Setup reads a dash

Reproduction

  1. nullius standard run agent-task-v1 reference --model <any> --limit 1 --unsandboxed
  2. Open the trial: the bar has no Setup segment and its chip reads "—".
  3. Read the trial's result.json: no agent_setup, and agent_execution starts where environment_setup ends.

Expected

A trial of an agent over ACP records its setup (its start and its session) apart from its run, as Harbor's agent_setup and agent_execution are. A start that fails is setup, not a run. The live board is told the same phases and seconds as the trial's result.

Acceptance

  • A standard trial of an agent driven over ACP records agent_setup, from the end of its environment's setup to its session opening; agent_execution and the trace's window start there.
  • A start that ends with no session (the agent never answered, exited, or the run was stopped) is recorded as setup, with no agent_execution.
  • A command agent, a capability, an Original setup and a control record no setup.
  • A task's internet door opens before the agent's phases and closes after them: neither is in agent_setup or agent_execution.
  • The live feed emits an agent_setup phase for such an agent, and its trial.end durations are the trial result's windows.
  • The field guide's phase rows hold for every runner, not Harbor alone.

Dependencies and related

The trial page's rework (#179, open; pull request #173) draws these phases and words their hovers. This is pull request #174. #173 merges first and words this phase's hover; #174 merges after it.

Activity

  1. added
    bugSomething isn't working
    P2Planned: limited in scope, has a way around it, or a missing feature
    on Nov 17, 2026
  2. added theissue type on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Planned: limited in scope, has a way around it, or a missing featurebugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions