Skip to content

Standard: a run cannot force its time limits, bound an agent's whole work on a unit, or cap its steps #190

Description

@aitianshi

Priority

extreme

Components

  • app/nullius (backend)
  • app/web (routes)
  • templates and static files
  • docs and the site
  • catalogue (benchmarks, agents, capabilities)
  • the standard (ports, kits, relay, runs)
  • CI, workflows and skills

Today

  • A standard run's time limits cannot be set by the user. Each phase has a fixed default (setup 600 s, image 3600 s, verify per task, score 600 s), and the agent's limit comes from the task or limits.turn_seconds.
  • The agent's limit applies to one reply, not to its work on the unit. On a conversation benchmark, τ³-bench's 600 s is per agent turn, so a conversation of 100 turns can take far longer than any single limit says.
  • Nothing bounds the steps of an agent that loops. A model with no price leaves no spending cap either.
  • A benchmark's own bounds are fixed and invisible in the builder: τ³-bench's --max-steps 200 and --max-errors 10, and each OSWorld agent's --max_steps (15, 50 or 100).

Wanted

  • Every phase's time limit can be forced for every participant, from the builder and from run. Empty keeps the default.
  • The agent's limit covers its whole work on a unit: every turn, and the simulated user's time between them.
  • Turns per conversation can be set, and only on a benchmark that holds conversations.
  • An optional cap on the model calls an agent makes on a unit, counted over the whole unit.
  • A benchmark's own bounds are options a run can set, shown in their own section of Limits.
  • An empty limit field shows the value the run will use: the benchmark's or the agent's own in the text colour, Nullius's default in grey (priority: Nullius < agent < benchmark).

Acceptance

  • run --timeout PHASE=SECONDS (setup, image, agent, verify, score) forces that phase's limit, and an unknown phase or a non-positive value is refused
  • The agent's limit is one deadline over the whole unit for an agent, while a capability keeps one limit per add and per recall
  • run --max-turns N sets a conversation's turns, and the builder offers it only when the benchmark needs agent.conversation/v1
  • run --max-calls N makes the relay refuse the agent's call past N with a 402. The unit ends as UnitCallCapError, classified as an agent error and not scored. Rate-limited or failed calls, and the judge's and the simulator's calls, are not counted. A capability run refuses it
  • config.json records the forced time limits, turns and calls under standard
  • τ³-bench offers Steps per conversation (1 to 200) and Tool errors per conversation, and its driver passes them to the upstream's run config
  • OSWorld offers Steps per task, which shows the chosen agent's own steps when empty and replaces or adds --max_steps when set
  • The builder's Limits step shows Scope, then Time limits in minutes, then a section named after the benchmark when it has bounds of its own
  • docs/cli/run.md, docs/standard.md, docs/builder.md and the field guide describe them

Dependencies and related

Activity

  1. self-assigned this
    on Nov 17, 2026
  2. added
    enhancementNew feature or request
    P0Fix now: unsafe, loses money or data, or nothing runs
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0Fix now: unsafe, loses money or data, or nothing runsenhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions