Skip to content

Standard: run each unit's agent in a Boat VM of its own #149

Closed

Description

@aitianshi

Priority

extreme. A new backend for runs of the standard: where an agent's commands execute.

Components

  • app/nullius (backend)
  • app/web (routes)
  • templates and static files
  • docs and the site
  • the standard (ports, kits, relay, runs)

Today

A standard run's agent runs either as the user on this machine (local processes, with an explicit yes to running unsandboxed) or in a container, which needs Docker or Podman here. A machine without either has no sandboxed way to run an agent.

Wanted

A third backend, Boat: each unit's agent or capability runs in a boat.dev VM of its own, started from this machine with a key saved in Settings → Sandbox. The VM's only way out is the unit's relay on this machine, so the agent never holds a model key. The VM is deleted whatever ends the unit. The builder's Where step offers it with a VM size, and greys out what the account cannot start.

Acceptance

  • nullius standard run ... --backend boat [--boat-size SIZE] runs each unit's agent or capability in its own Boat VM, and conform takes --backend boat
  • The agent in the VM reaches only its relay; no model key is sent to the VM
  • Every VM a run starts is deleted, on success, failure, timeout and Stop
  • The VM's auto-stop follows the unit's time limit, within the 2 hours a free trial allows
  • Time spent starting the VM and installing the agent is not counted as the agent's time
  • Nothing under the user's own private folders (home, keys, Nullius config) is sent to a VM through a link
  • Settings → Sandbox saves, edits and removes the Boat key; Remove asks first
  • The builder's Where step offers Boat only on this machine, with the account's VM sizes; Review refuses a size the account cannot start
  • A run says when the account can start fewer VMs this hour or today than the run has units
  • Units that need a benchmark's MCP server or their own image end as not run on Boat, with the reason
  • docs/standard.md, docs/builder.md and docs/cli/run.md describe the backend

Dependencies and related

Activity

  1. self-assigned this
    on Nov 16, 2026
  2. added
    enhancementNew feature or request
    P2Planned: limited in scope, has a way around it, or a missing feature
    on Nov 16, 2026
  3. aitianshi commented on Nov 17, 2026

    @aitianshi
    CollaboratorAuthor

    Scope update from the review of #150:

    • On Boat, a check that executes the agent's code (score.runs_code: HumanEval, the workspace kit) runs in this machine's throwaway check container, so such a run needs Docker here and is refused without it.
    • A unit whose setup gives the agent MCP servers (the tools kit, τ³-bench) is not run on Boat: they start from this machine's paths.
    • The Boat key is withheld from every other command Nullius runs, and a VM Boat will not delete is reported (delete_error in agent/boat.json).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Planned: limited in scope, has a way around it, or a missing featureenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions