Repository navigation
feat(standard): run each unit's agent in a Boat VM of its own - #150
Merged
Merged
Conversation
This comment has been minimized.
This comment has been minimized.
Adds the boat backend (app/nullius/standard/boat.py): one boat.dev VM per unit, the agent as a user locked to the VM's loopback, its model calls through a reverse tunnel to the unit's relay on this machine, the workspace copied back without links that leave it, the VM deleted after. Concurrency is capped to the VMs the account runs at once. The builder's Where step offers Boat; Settings > Sandbox keeps its API key with Edit and Remove.
…ot start every unit Send the install before the agent's time and the home and workspace as it starts; follow the unit's time with the VM's auto-stop, within a trial's 2 hours; refuse links that would send this user's own files; delete a VM whatever ends the unit, and skip the fetch on a Stop. Review refuses a size the account cannot start, Remove asks first, the run says when the account can start fewer VMs this hour or today than it has units, and the Boat mark has its own version on a dark background.
- A Boat run whose check executes the agent's code checks it in the throwaway check container, and is refused without Docker; it ran as the user on this machine. The workspace kit's check declares runs_code. - A unit whose setup gives the agent MCP servers is not run on Boat: they start from this machine's paths, which the VM does not have. - The agent's user's firewall refuses DNS before it accepts loopback; the rules were inserted in an order that let DNS through. - The agent's environment (the relay's word, a licence's value) goes to the VM as a file its user reads and removes, never on an ssh command line. - A Stop ends the wait for a VM to come up, and the VM is deleted. - Runtimes linked from ~/.local, ~/.cache, ~/.config and other tools' login folders are refused, with the Backend's own environment. - The Boat key is withheld from every run's environment and handed over stdin only to a standard run on Boat. - Once a trial's 2-hour cut is known, later units ask for it at once.
A delete is tried twice, and a VM already gone counts as deleted. One Boat still refuses is said, with when it stops by itself, and kept as delete_error in the unit's agent/boat.json; the end-of-run sweep says each VM of the run it could not delete.
aitianshi
force-pushed
the
research/boat-sandbox-fit
branch
from
November 17, 2026 15:39
f1368d9 to
bf70b27
Compare
This comment has been minimized.
This comment has been minimized.
11 tasks
The section's table was the mechanism, cell by cell. It now keeps the guarantees, the requirements and the refusals; the builder, CLI and Settings rows say only what Boat is and where it is set.
This comment has been minimized.
This comment has been minimized.
Why a backend cannot be used sat beside its name without wrapping, which squeezed the name until it broke mid-word (Containe/r) once Boat made the row three tiles. It now goes under the name; a row's tiles stay one height.
This comment has been minimized.
This comment has been minimized.
A run on a host starts its Boat VMs from there. The key stays in this machine's store and reaches the run over stdin, as the agent keys do; Review checks it here, and the host's ssh, ssh-keygen, rsync and tar among its prerequisites. A Boat run whose check runs the agent's code now asks Review for Docker where it runs.
This comment has been minimized.
This comment has been minimized.
The Boat section drops what is mechanism (how the key travels, what a host is checked for) and keeps what a user decides on. AGENTS.md gains a Documentation convention so later changes keep the same standard.
This comment has been minimized.
This comment has been minimized.
A run started by Nullius keeps only its provider's key and endpoint of the environment its launcher gives it (run_env), which dropped the Boat key handed over stdin. Here the run fell back to the key saved in Settings; a host has none, so a run on Boat there was refused for want of a key.
🔴 PR judge · 5 of 8 yes · #150 ↔ #149
Acceptance criteria, 11 of 11 met
Checks the script runs itself, 4 of 4 pass
some of the diff was cut before the judge read it |
aitianshi
added a commit
that referenced
this pull request
Nov 17, 2026
… work on a unit and cap its model calls (#191) Closes #190 ## What A standard run can now force its limits, and the agent's time limit covers its whole work on a unit. - **Time limits by phase.** `run --timeout PHASE=SECONDS` forces the limit of a phase: `setup`, `image`, `agent`, `verify` or `score`. Empty keeps the default, which comes from the task, then the benchmark, then Nullius. `config.json` records what was forced under `standard.timeouts`. - **One agent limit per unit.** The agent's limit is one deadline for its whole work on a unit: every turn of a conversation, and the simulated user's time between them. Before, it applied to each reply. A capability keeps one limit per add and per recall, because a memory adds a unit's sessions one by one. τ³-bench's `limits.turn_seconds` goes from 600 s a reply to 7200 s a conversation. - **Turns.** `run --max-turns N` sets the turns of a conversation. The builder offers it only on a benchmark that holds conversations. - **A cap on model calls.** `run --max-calls N` caps the model calls an agent makes on a unit. The relay counts each answered call on the agent's route; rate-limited and failed calls are not counted, and neither are the judge's or the simulator's. Past the cap the relay refuses the call with a 402, and the unit ends as `UnitCallCapError`, an agent error that is not scored. It is for agents only: a capability's route also carries the benchmark's reader, so a capability run refuses it. - **A benchmark's own bounds.** They are now options marked `limit`. τ³-bench gets Steps per conversation (`--max-steps`, from 1 up to its default of 200) and Tool errors per conversation (`--max-errors`). OSWorld gets Steps per task (each agent script's `--max_steps`); left empty it uses the chosen agent's own value, declared with the new `by` key. - **Builder, Limits step.** It shows Scope, then Time limits in minutes, then a section named after the benchmark when it has bounds of its own. An empty field shows the value the run uses: in the text colour when the benchmark or the agent declares it, grey when it is Nullius's default (priority: Nullius < agent < benchmark). Nothing is typed into a field for the user. ## Touches - `app/nullius/standard/`: `run.py`, `relay.py`, `container.py`, `job.py`, `cli.py`, `recipe.py`, `definitions.py` - `app/nullius/runs/classify.py`: `UnitCallCapError` is an agent error - Catalogue: τ³-bench and OSWorld's `standard.json`, their drivers and `upstream-settings.json` - `app/static/builder/`: the Limits step - Docs: `docs/cli/run.md`, `docs/standard.md`, `docs/builder.md`, the field guide and the generated supported-benchmark pages ## Before and after The Limits step for τ³-bench, before and after: | Before | After | | --- | --- | | <img width="440" alt="Limits step on main: Scope only, with tasks, concurrency, attempts and two spending caps" src="https://github.com/user-attachments/assets/80b468a4-326f-4057-a4b2-5551ea93e642" /> | <img width="440" alt="Limits step on this branch: Scope with turns and model calls, Time limits in minutes, then a tau3-bench section with steps and tool errors" src="https://github.com/user-attachments/assets/6278494c-6c9f-456f-b739-9578494315ee" /> | ## Verified - Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging `main`, with #124's serving providers and #150's Boat VMs). - New tests: - A whole conversation that ends past the agent limit, though no single turn does. - Four one-call turns cut on the fourth call by `max_calls=3`. - The relay counts only answered calls on the agent's route. - A slow capability whose three adds each fit a 1 s limit while their total does not. - A capability run refuses `--max-calls`. - Recipe validation of the limits, including a value that rounds to 0 s. - τ³-bench steps above 200 are refused. - OSWorld's `--max_steps` is replaced, or added when the agent's command doesn't set it. - Against the installed τ³-bench upstream: the driver uses 200 steps and 10 errors by default, and 7 and 3 when the run sets them. - Checked in the browser, light and dark, for τ³-bench and OSWorld. ## Risk - **Agent timeouts on conversation benchmarks.** A conversation that used to fit 600 s per reply now has a single limit for the whole conversation, 7200 s on τ³-bench. A very long conversation can now time out where it didn't before. Undo by forcing a larger `--timeout agent=SECONDS`, or by reverting `limits.turn_seconds`. - **Comparability.** A run whose limits differ from the defaults is not yet marked as not comparable with the leaderboard. - **Overlap.** #174 (the agent's setup timed apart from its run) and #133 (a benchmark's own limits) change the same parts of `run.py` and the catalogue, so expect merge conflicts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #149
What
Adds Boat as a third backend for runs of the standard. Each unit's agent or capability runs in a boat.dev VM of its own, started where the run runs (this machine or a Settings host) with the key saved here in Settings → Sandbox.
How a unit runs on Boat
ssh -Rtunnel, so the VM never holds a model key.--safe-links.score.runs_code: HumanEval, the workspace kit) runs it back where the run runs, in the throwaway check container, never as the user. Review asks for Docker there, and the run is refused without it.delete_errorinagent/boat.json.Refusals and warnings
~/.config,~/.local,~/.docker, …) or Nullius's storage to a VM is refused.The key
ssh,ssh-keygen,rsyncandtaramong its prerequisites.Interface
Touches
backend · routes · templates and static · docs and site · the standard
app/nullius/standard/boat.py(new),run.py,recipe.py,job.py,cli.py,kits/agent-workspace-v1/standard.json(runs_code)app/nullius/execution.py(the Boat key withheld, handed to a Boat run),prerequisites.py(a host's Boat programs)app/web/execute.py,builder.py,settings.pyapp/static/builder/,app/static/settings/sandbox.js,app/templates/settings/settings.html,app/static/style.css,app/static/brand/boat.svgandboat-dark.svg(Boat's own marks, copied unchanged)app/tests/test_standard_boat.py,app/tests/test_execution.py,app/tests/test_prerequisites.py,app/ui_tests/test_settings_page.py,app/ui_tests/test_agents.pydocs/standard.md,docs/builder.md,docs/cli/run.md,docs/settings/sandbox.md;AGENTS.md(a Documentation convention:docs/stays terse)Before and after
Screenshots were taken locally, in light at 1280 px and dark at 390 px:
Verified
Tests, on
5cd922b(main) plus this branch:test_standard_boat.py: 36 tests with a stand-in VM. Besides the run itself they cover:~/.localand~/.configincluded;test_execution.py: only a standard run on Boat is handed the Boat key.Real runs on a free-trial Boat account
meta/muse-spark-1.3-contributor,--max-usd 0.10.Before the review fixes, every agent scored 2/2:
After the fixes, and again after the rebase onto main: opencode scored 2/2 three times. In the two runs whose trials were read, each recorded
check: container.dns=2);internet=7);loopback=0);iptables -S OUTPUTlisted the DNS rejects before the loopback accept.GET /sandboxesreturned[]). Model spend was about $0.02 in all.Browser
Not verified
Risk
run.py: the unit's VM context and the agent-time ordering.runs_code. That changes nothing on the local or container backends; it only makes a Boat run of the kit need Docker.EnvironmentSetupErrorand the other backends are unaffected.