Repository navigation
feat(catalogue): add OSWorld2 with OSWorld-V2's own agents, its services on the machine and a null control - #134
sirotaoleg wants to merge 8 commits into
Conversation
Closes #136 ## What On Runs → By launch, an agent or control with 2 or more shards is now one row, when `cli merge` would merge those shards. The row shows the numbers of the merged job. `launches.fold` makes that job in memory: the trials of every shard, and the job record from `merge.result_of`. `result_of` is the code that `merge` used inline to compute the job record, now a shared function. `merge.check` keeps the trial records that it already reads. - The rate is over every trial with a verdict, never an average of shard rates. - Units that no shard ran come from `merge.check` and stand beside the count ("2 of 8 not run"). They are never misses. The notes of `merge` are the tooltip. - The state is the most telling state of the shards, with a warning such as "1 of 2 shards stopped". - The header gives the dataset size, not the size of one shard. The row is folded by default. A click on the row, or Enter or Space on its chevron, unfolds one row per shard ("shard 2 · host-2"), each with its Open. The fold uses the shared chevron and slide of `disclosures.js`, and respects reduced motion. A refresh of the list keeps an unfolded row unfolded. Controls fold the same way, stay unranked and stay out of Compare. The task-check line is judged over every shard: "Task checks incomplete: oracle not run on shard 2." A participant with one shard keeps its plain row. Shards that `merge` refuses also keep their rows. Job pages, `cli merge` and merged jobs do not change. The header counts every shard ("8 runs"), every shard has its row, and each folded row says how many shards it holds ("2 shards"). A fold reads every trial record of its shards. It is kept in memory for the shard jobs that the workspace cache gives, and it is read again only when the workspace parses a shard again. A new set of shards replaces each kept fold that shares a shard with it, so the cache holds one fold per shard at most. ## Touches backend · the standard (`merge.py`) · routes · templates and static · docs and site ## Before and after One launch over 2 machines, 8 units: 2 agents, nop and oracle. Shard 2 stopped while terminus-2 ran (2 of its 4 units), and the oracle did not run on shard 2. Light theme, 1280 px. | Before | After, folded | | --- | --- | | <img width="980" height="670" alt="1-before-shards-apart" src="https://github.com/user-attachments/assets/990ceda8-8212-4003-ab1c-f380be783fc9" /> | <img width="980" height="421" alt="2-after-folded" src="https://github.com/user-attachments/assets/f37d9af1-7b3b-4b05-abc6-7dca2da1c983" /> | Before: 8 rows, 2 rates for each agent, "4 tasks", "oracle not run". After: 4 rows, "8 tasks". terminus-2 shows 50% and "3/6 · 2 of 8 not run" with "1 of 2 shards stopped". The task-check line reads "Task checks incomplete: oracle not run on shard 2." | After, unfolded (click and keyboard) | After, dark, 390 px | | --- | --- | | <img width="980" height="595" alt="3-after-unfolded" src="https://github.com/user-attachments/assets/b30f3189-d737-4658-8701-a3bbc434ad4a" /> | <img width="358" height="844" alt="4-after-dark-390" src="https://github.com/user-attachments/assets/6c4b106c-218b-4308-a934-ecac1f96934d" /> | ## Verified - Backend test `Fold.test_a_fold_reads_as_the_merged_job`: three shards, one absent, and one with a planned unit that did not run. The folded job equals `parse_job` of the `merge.merge` output on every field the row shows (solved, rates, outcomes, planned, errors, start, end, time, tokens, cost, mean score, budget). The rate is not the mean of the shard rates. The units not run and the notes are those of `merge`. Shards that `merge` refuses are not folded. - Backend test `Fold.test_a_fold_is_read_again_only_when_the_workspace_parses_a_shard_again`: a second fold of the same cached jobs does not call `merge.check`. After a trial record changes, it calls `merge.check` again. After a third shard appears, the cache holds only the fold of the three. - Route tests `test_launch_shards`: 8 shards give "8 runs", 8 shard rows and 4 folded rows that each say "2 shards". The rows are folded by default. Each shard row has its Open, and the folded row has none. The rate and time of the folded row equal those of the merged job. "2 of 8 not run" and "1 of 2 shards stopped" show. The controls fold and stay out of Compare. A nop that passes one task on shard 2 gives "A control did not behave". A stopped nop shard gives "nop stopped on shard 2". Participants with one shard keep their plain rows. - On `c7af06d5`: backend suite `python3 app/tests/parallel.py` (1989 tests) and route suite `python3 app/tests/parallel.py app/ui_tests` (328 tests) pass in a clean container. - Browser, with the fixture above: folded and unfolded, light and dark, 1280 and 390 px. No page errors and no horizontal page scroll. A click on the row unfolds it, and the shard rows slide open and closed. Tab moves the focus to the chevron of the oracle, Enter unfolds it, and Space folds it again. With reduced motion, the rows show at once. At 390 px, a long agent name wraps and the chevron stays on the line of the name. The name, its "2 shards" line and the shard rows start at the same left edge. - Render time of `/runs?view=launch`, median of 15 requests after a first one, with the same parsed jobs given to the page. 2 agents and nop and oracle, each 4 shards of 500 units (8000 trials): 71 ms before this change, 101 ms after. The first render reads the trial records once: 540 ms, against 199 ms before. The rest of the added time is the outcome bars of the 4 folded rows, over 2000 trials each. 8 shards of 1 unit each (32 trials): 11 ms before, 13 ms after. ## Risk If the fold is wrong, a row shows numbers that differ from the merged job. The backend test compares the two directly. `merge` writes the same `result.json` as before through `result_of`, and its tests pass unchanged. The first render of a large sharded launch reads all its trial records once (+0.3 s for 8000 trials). Later renders use the cache until a shard changes. The cache holds one fold per shard at most, with the parsed jobs that the workspace cache also holds. Open pull requests #124, #133 and #134 also change `merge.py`: the second to merge must rebase. Revert the commit to undo.
…l everywhere (#148) Closes #147 ## What Each model provider is now one record that every feature reads. A feature asks for the protocol that it speaks, and gets the providers that serve it. - **The registry** (`app/nullius/providers.py`). `PROTOCOLS` defines each wire format once: its default path and how the key is sent (`chat`, `responses`, `messages`, `systemone`, `decisions`, `openai-decisions`). `PRESETS` holds OpenRouter, OpenAI, Anthropic and TypeSafe. `record()` returns `{name, base, key, endpoints}` for a preset or a custom provider. `key` is the name of the secret setting, never the key. - **Storage** (`app/nullius/settings.py`). A record is one JSON setting of a new kind, `provider`. A custom provider is `MODEL_<ID>`. The endpoints added to a preset are `OPENROUTER`, `OPENAI`, `ANTHROPIC` or `TYPESAFE`. For these settings the environment does not replace the file. Its value is a partial object, merged over the file's value (RFC 7396, `null` removes a field). Settings shows read-only only the fields that the environment sets. Keys keep their names and stay separate secrets. - **Migration.** `MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` are still read as `name` and `base`. Saving a card moves them into the record. `nullius migrate --providers` previews that rewrite of `settings.json`, and `--apply` writes it. Recipes do not change. `local` and `local:<ID>` still read as `custom`. - **Consumers.** The registry is read by the builder, the provider test, onboarding, `execution`, standard runs, the relay, the CLI's `--provider` and both judges. The builder's JavaScript copy of the list is gone. The builder offers a provider only where it serves what the recipe's agents and judge speak, so Anthropic appears for an agent of the standard that speaks `messages`. Review, Run (`web/execute.py`) and `run.Run` refuse a provider that does not fit. The relay sends `x-api-key` and `anthropic-version` on a `messages` route only, and a bearer header on a benchmark's own routes. - **Settings cards.** A card shows Base and a collapsed Endpoints section with one row for each protocol. The placeholder is the default URL. A base on a preset's host suggests that preset's endpoints. A preset's known endpoints are fixed. An endpoint added to a preset is a path, or a full URL on the preset's own host, because the preset's key goes there. - **System One** (`app/nullius/judge/client.py`) asks each decision protocol: the TypeSafe shape on `systemone` and `decisions`, and its own adapter for `openai-decisions`. It keeps the provider's `usage.cost` and records a `refusal` as no answer. A 402 says what it means. A judge never follows a redirect with the key. - **Judge cards.** Each judge has Provider, then Protocol when the provider serves two protocols of that judge (else one quiet line, "via ..."), then Model. Each judge's provider is a select whatever the count, because the list is the user's own providers and grows or shrinks with them; cards are kept for fixed sets of choices. New settings: `LLM_JUDGE_PROTOCOL`, `SYSTEM_ONE_PROTOCOL`. On OpenRouter, System One offers OpenRouter's public list of decision models, asked without a key, cached 6 hours, with the fixed list as the fallback. The seven commits are atomic and in reading order: registry and storage, consumers, Settings cards, decision protocols, judge cards, tests, docs. ## Touches backend (`providers`, `settings`, `migrate`, `onboarding`, `execution`, `judge/client`, `recipes/launch`) · the standard (`run`, `relay`, `recipe`, `cli`, `definitions`, `fidelity`) · routes (`web/settings`, `web/execute`, `web/filters`) · templates and static (Settings cards, judge cards, builder core) · docs (`docs/settings/model-providers.md`, `system-one.md`, `llm-judge.md`, `README.md`, `agent-keys.md`, `docs/builder.md`, `docs/judgement.md`, `docs/standard.md`, `docs/cli/README.md`, `AGENTS.md`, `app/.env.example`) ## Before and after Settings at 1280 px, light theme. Fixture keys only. Model providers. TypeSafe is a preset like the others, so it comes after the run providers. | Before | After | | --- | --- | | <img width="748" height="431" alt="1-before-providers" src="https://github.com/user-attachments/assets/1dd2338e-0b32-4631-8895-a6df2ba7fc29" /> | <img width="748" height="431" alt="2-after-providers" src="https://github.com/user-attachments/assets/ff3bb210-22ef-4bff-bbe4-df4ac4488519" /> | A preset card (OpenRouter), Endpoints open. Known endpoints are fixed. Other protocols are editable rows. | Before | After | | --- | --- | | <img width="748" height="360" alt="3-before-preset-card" src="https://github.com/user-attachments/assets/5fb776e9-1c86-4cf0-8508-24041b6f9042" /> | <img width="748" height="706" alt="4-after-preset-card" src="https://github.com/user-attachments/assets/9058e8e4-339e-49da-8a44-67b2ffcdf8bf" /> | A custom provider, Endpoints open, with one endpoint set. | Before | After | | --- | --- | | <img width="748" height="451" alt="5-before-custom-card" src="https://github.com/user-attachments/assets/c725f642-3d22-4b8d-8fb0-c702e415594d" /> | <img width="748" height="897" alt="6-after-custom-card" src="https://github.com/user-attachments/assets/afcbcb01-fed9-4ef3-87c2-883d73594edc" /> | Judges. System One on OpenRouter serves two of its protocols, so it shows Protocol. The LLM judge on OpenAI serves one, so it shows "via OpenAI Chat Completions". | Before | After | | --- | --- | | <img width="748" height="504" alt="7-before-judges" src="https://github.com/user-attachments/assets/a089a1bd-9f72-4ca6-bff1-e1a652a83ec1" /> | <img width="748" height="503" alt="8-after-judges" src="https://github.com/user-attachments/assets/61ef9ae3-e080-4e54-99a1-dc43d4960ba6" /> | The custom card at 390 px: <img width="358" height="1153" alt="9-after-custom-card-390" src="https://github.com/user-attachments/assets/dce35d02-44b1-4ccb-8362-c84129e1eb87" /> ## Verified The branch is based on `main` at 1cc37ac. I tested its tip, each suite in a new rootless Podman container (`python:3.12`, `--init`, uv on PATH, Node for the route suite): - Backend suite (`python3 app/tests/parallel.py`): 2029 tests in 118 modules, OK. `main` at 1cc37ac, in the same container: 1992 tests in 116 modules, OK. - Route suite (`python3 app/tests/parallel.py app/ui_tests`): 441 tests in 42 modules, OK. `main` at 1cc37ac: 431 tests in 41 modules, OK. - New modules: `test_provider_registry` (17 tests: the merge, `null`, partial environment values, the old settings, record checks, the preset host rule, keys never in a record), `test_judge_decisions` (18 tests: each decision protocol's request and reply, `usage.cost`, `refusal`, 402, no redirect, the cached model list and its fallback), `ui_tests.test_judge_cards` (2 tests: the protocol row and the live list). Existing modules gained tests for the relay's headers, the builder's provider filter, Harbor recipes on Anthropic and the Settings cards. - End to end, against the server (`app/app.py`) and the CLI, in a container with no network, with a new `NULLIUS_HOME` for each case and fixture keys only. 37 checks, all pass: - No settings: the four presets and the OpenAI-compatible card. - A custom provider in `settings.json`: its card, its merged record, the default path for a protocol without an entry, System One offers it for its `decisions` endpoint, no key in the page. - `MODEL_GW='{"base": "https://gw-eu.example/v1", "endpoints": {"decisions": null}}'` in the environment: the base changes, the name stays the file's, the endpoint goes, System One no longer offers it. The card shows Base read-only and Name editable. The file does not change. - `nullius migrate --providers`: the preview names the moves, prints no value and leaves the file as it was. `--apply` writes the record, removes the old lines, keeps the key and the file mode 0600. A second run finds nothing to move. - Old variables in the environment (`MODEL_API_URL`, `MODEL_<ID>_NAME`, `MODEL_<ID>_API_URL`, `LLM_JUDGE_PROVIDER=local:<ID>`): read as before. - System One's model list on OpenRouter, offline: the route answers with the built-in list in less than the 5 s timeout, or with a cached list, however old. - Browser (Chromium, Playwright), on this branch and on `main`, at 1280 px and 390 px, light and dark, with no network: the provider list, a preset card and a custom card with Endpoints open, and both judge cards. No page errors and no horizontal scroll. Not verified: live calls to OpenRouter Decisions and OpenAI Decisions, and a whole run on Anthropic. The tests use fixtures. ## Risk A wrong record or header can send a call to the wrong URL, or with the wrong key header. Undo with a revert. The old settings are still read, so a revert keeps working with them. Records written by a card stay in `settings.json` and the older code ignores them. - **Migration on edit.** Saving a custom provider's card moves `MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` into `MODEL_<ID>`. After a revert, that provider loses its name and base until they are set again. `nullius migrate --providers` without `--apply` shows what would move. - **New reserved names.** `OPENROUTER`, `OPENAI`, `ANTHROPIC`, `TYPESAFE` and `MODEL_<ID>` are now settings. An agent key listed under one of these names is left out, and Settings says so. A shell variable such as `MODEL_PATH` that is not a JSON object is not read as a record. - **`--provider anthropic` on the CLI.** The key header now follows the agent's protocol. An agent that speaks only `chat` is refused on Anthropic, where it used to get `x-api-key` on OpenAI-style paths. - **Runs and custom endpoint paths.** The relay forwards the agent's own path below the provider's base. An Endpoints row changes where the judges and the test ask, not where a run's calls go. - **`openai-decisions` field names** come from OpenAI's documentation, not from a live reply. - **Open PRs on the same files.** This change does not depend on any of them. Whichever merges second rebases. A test merge of this branch with each one: - #124 (OpenRouter serving providers, `--served-by`): conflicts in `providers.py`, `standard/run.py`, `standard/relay.py` and `docs/settings/model-providers.md`. Each conflict is small. Both add functions at the same place in `providers.py`. This branch removes `run.PROVIDERS`, where #124 adds `routing` to the OpenRouter entry: that flag becomes a field of the OpenRouter preset. Both change `Relay._headers`. - #133 (stacked on #124): the same four files, and also `standard/fidelity.py` (one `key_header=` line, now `r.key_header`) and `docs/standard.md` (one table row each). - #134 (stacked on #133): the same as #133, and also `ui_tests/test_settings_page.py` (one placeholder assertion, which both change). - #124, #133 and #134 also conflict with `main` alone in `standard/merge.py` (and #133, #134 in `tests/test_standard_shard.py`), which this branch does not touch.
…on top of --served-by: gated tasks, a definition's pins as its serving pins, a pinned route's refusal as the unit's, limits of their own, and prove of a merged control A definition's pin (setups.original.pin, score.judge.pin) joins the run's --served-by pins and wins over --served-by for its model; ApiProviderRouteError reads the pin and reply.served_by. A failed Responses object reads reply.end and reply.error.
…eep the score of a unit a guard stopped despite a last 429 limits.run_again false: a unit that ended rate-limited is not run once more; its outcome stands. A unit whose driver scored it after a guard's stop keeps that score, marked stopped, though its last own call got a 429.
… own last 429, not a side call's or a context window's stop The rule reads the agent's own last call, a guard's line in the stop file and a score stopped by other than the context window; a cap's refusal is still the cap's. limits.run_again takes null as unset. The rate limits row names the opt-out.
… upstream's own runners, mock websites, GitLab and proxy, proven on test_v2, and how to prepare a machine for it OSWorld2 runs each of the 108 tasks of test_v2 in a VM of its own through the upstream's docker provider, with the mock websites its tasks need served on the machine, and GitLab and a proxy started for each unit that needs one. Its Original runs the upstream's agents with their models pinned to their providers on OpenRouter, and its null control is proven on test_v2 at 0 of 108. The tasks are gated, so a failing endpoint is kept as its status alone. Its machine page says what a machine needs, how to check it and what a run costs.
…reach OSWorld2's VMs and services
limits.run_again false: an OSWorld2 episode is paid and long; one that ended rate-limited is run again by hand.
…g programs, and fake the sweep's clock for the sweep alone Two tests made a Run of OSWorld2, which asks for ImageMagick, ffmpeg and tesseract on the machine: they now say the programs are there, as the test of their refusal does. The sweep's tests replaced time.sleep for every module, so a slow docker's wait in subprocess advanced the fake clock and the test's own steps: the fake now replaces the sweep's time module alone.
da7318f to
592be82
Compare
aitianshi
left a comment
There was a problem hiding this comment.
GitLab for tasks 026 and 041 should be supported automatically, or left out of this PR.
Right now it is half done. Each unit starts GitLab from its snapshot and removes it afterwards, which is automatic. The one-time preparation is not: docs/benchmarks/osworld2-machine.md step 9 asks the user to clone Task-Web/gitlab at the pinned commit, run gitlab_server.py configure, compose pull and init, choose a root token, paste it into Custom keys as GITLAB_PRIVATE_TOKEN, and set GitLab host. A user should not have to set up a GitLab server by hand to run a dataset.
The builder makes this visible on every card. It shows Add GITLAB_PRIVATE_TOKEN in Custom keys (some tasks may need it). even on "Offline, no services", whose 36 tasks never touch GitLab. That happens because no dataset names an environment, so credentialWarnings treats the gitlab environment's secret as possibly needed everywhere.
Please choose one of these:
- Support it automatically. Nullius prepares GitLab on the run machine as part of the run's prerequisites (
nullius/prerequisites.py): clone at the pin, configure, pull,init. It generates the root token there and keeps it on that machine, so the user never types or pastes one. The builder then shows nothing about GitLab unless the preparation fails. - Remove it from this PR. Drop the GitLab parts (
gitlab_server.py,gitlab.compose.yml, thegitlabenvironment and option, the licence, step 9 of the doc, its tests). Tasks 026 and 041 then stay out of the offered datasets until a follow-up PR supports them fully.
Either way, a selection that cannot need the token should not warn about it.
| "good_to_know": [ | ||
| "Scores are partial credit: each task's own evaluate() gives 0 to 1, and the upstream's score is the mean." | ||
| ], | ||
| "limitations": [ |
There was a problem hiding this comment.
Limitations are far too long. Please cut them down a lot.
There are 25 items and about 2,200 words; one item alone (the LLM grader) is 332 words. On OSWorld2's page they fill several screens, and nobody will read them before choosing a run.
The UI rules (.claude/rules/ui.md) say Good to know and limitations must change a user's choice, the action they have to take, or how they read the results. They also say to leave out dependency and installation details that Nullius handles itself, and that removing text beats adding it. Most of these items are how the driver works inside, how errors are classified, or how the machine is set up:
- installation and machine setup: mock websites build, GitLab, proxy image, VM resources, write limits, Settings constraints, output kept per trial. These belong in
docs/benchmarks/osworld2-machine.md, or nowhere if Nullius does them; - outcome mechanics: how a setup failure, a timeout, the upstream's own failures, the 10-hour stop, a grader refusal or a context overflow is scored. These belong in the docs or in the trial's own record.
What a user actually needs is roughly this, one short sentence each:
- not the paper's setup: another release, no residential proxy, and 100 steps with thinking off by default (Steps 500 and Thinking adaptive restore the budget);
- Muse Spark's published score came from newer upstream code and does not compare;
- the grader and the simulated user are Claude Sonnet 4.6 on OpenRouter: with another provider those tasks fail;
- tasks that use the public internet can change outcome with the live sites;
- tasks 103 and 104 are not supported;
- a slower machine can lower scores;
- the null control scores above 0 on five tasks.
Aim for under 10 items, each one line.
|
The screenshots in the description can't be read without opening them, and opening them fails. Each image is shown at Please put the screenshots directly in the description at full size, for example by dragging them into the editor and leaving GitHub's own |
| "websites": { | ||
| "label": "Website host suffix", | ||
| "summary": "WEBSITE_HOST_SUFFIX: none, or the host suffix OSWorld-web's sites are served under for this machine's VMs, a wildcard DNS name of the address the stack listens on, the gateway of the VMs' own network (10.213.0.1.nip.io). The 40 tasks that use the sites need it.", | ||
| "pattern": "none|(?=.{3,200}$)[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?:\\.[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?)+", |
There was a problem hiding this comment.
The page should show an example for these options, not the regex.
OSWorld2's page prints the raw pattern of all five free-text options (Website host suffix, GitLab host, GitLab write share, Disk write MB/s, Disk write IOPS). It shows none|(?=.{3,200}$)[a-z0-9](?:[a-z0-9-]{0,61}... where a user needs to see what to type. catalogue/benchmark.html only hides the pattern for temperature ("Custom decimal, e.g. 0.5"); every other pattern option falls through to the raw regex. Before this PR no other option had a pattern, so nobody saw it.
Suggested:
- each pattern option carries an example, e.g.
"example": "10.213.0.1.nip.io","gitlab.10.213.0.1.nip.io","16:160","16","160"; - the page shows "e.g.
16:160" instead of the pattern, the same way temperature already does, so the special case goes away; - the pattern stays for validation only;
- the summaries drop the internal names they start with (
WEBSITE_HOST_SUFFIX:,GITLAB_URL:,device_write_bps), following the "Names, not keys" rule in.claude/rules/ui.md.
| }, | ||
| "websites": { | ||
| "label": "Website host suffix", | ||
| "summary": "WEBSITE_HOST_SUFFIX: none, or the host suffix OSWorld-web's sites are served under for this machine's VMs, a wildcard DNS name of the address the stack listens on, the gateway of the VMs' own network (10.213.0.1.nip.io). The 40 tasks that use the sites need it.", |
There was a problem hiding this comment.
Please make the option and setup descriptions much shorter.
On the page, each option's description runs 2 to 4 lines (35 to 44 words for Website host suffix, GitLab host, GitLab write share and Disk write MB/s). The two Setup blocks are longer still (Docker is 54 words). They explain how things work inside: Caddy, snapshots, device_write_bps, 1024 * 1024 bytes, the network nullius-osworld2. A user only needs to know what the value does and when to set it. The UI rules (.claude/rules/ui.md): "Less text", "Removing text beats adding it", and the details go on the entry's own docs page.
For example:
- Website host suffix: "Where the mock websites are served. Needed by the 40 website tasks."
- GitLab host: "Where GitLab is served. Needed by tasks 026 and 041."
- GitLab write share: "Disk write limit for GitLab, as MB/s:IOPS."
- Disk write MB/s / IOPS: "Disk write limit for each VM."
- Setup: Docker: "Docker without sudo; KVM recommended."
- Setup: scoring programs: "ImageMagick, ffmpeg and tesseract, used to score 6 tasks."
Two related things on the same page:
- the Setup headings show raw keys (
Setup: docker,Setup: scoring_programs) becausebenchmark.htmlprints the key. They need a label, per "Names, not keys"; - the
thinkingoption has nolabel.
Stacked on #133: this pull request's own commits are the last five. Until #133 is merged, the diff shows its files too. It can be merged after #133.
What
OSWorld-V2's 108 tasks become a bundled benchmark,
osworld2, on the Nullius standard, at the upstream's releaseosworld-v2.1. Each task is a unit in an Ubuntu VM of its own on the upstream's docker provider, set up by the task's own class and scored from 0 to 1 by its own evaluate(). The tasks and their assets are gated on Hugging Face. Nullius does not fetch them, and the entry isgated(#133): a unit's failure from a model endpoint says its status alone, and the driver names an upstream exception by a known class's name at most, never by its words. The Original setup runs one of OSWorld-V2's own agents, M3 or Muse Spark, chosen by the Agent option, from its published script, with every model call sent to the relay. The Original's models are pinned to their providers on OpenRouter: M3 to MiniMax, Muse Spark to Meta. Claude Sonnet 4.6 is pinned to Anthropic; it is the LLM grader of 21 tasks and the simulated user of 6. Only M3's runs count as comparable with the paper's row. Its Review row and its generated page name the setup "OSWorld-V2 agents". A null control runs oneDONEaction, then the task's evaluate() on the untouched VM. The entry declaresagent.computer/v1. No agent from the Agents page runs it yet. Its texts are one Good to know line and 25 limitations. Settings lists GitLab's root token under Custom keys, for the two GitLab tasks.The mock websites of 40 tasks are OSWorld-web at the same tag. The guide's steps build them from source and serve them on the machine the units run on. A unit that needs them is refused before its VM starts when they do not answer. For 026 and 041, the driver starts GitLab anew from its snapshot, one task at a time, and Overleaf's 2 tasks get Overleaf started anew. Each of the 8 tasks that ask for a proxy gets a proxy of its own. The options are Steps (100, 300 or 500), Thinking for M3, the website host suffix, the GitLab host and its write share, and disk write limits for each VM. By default a run takes 100 steps with thinking off, as the upstream's M3 script does; the paper ran M3 at 500 steps with thinking on. Muse Spark runs as published at this release, while its published score came from newer upstream code; a limitation says what differs. There are six datasets: All tasks (108, the upstream's
test_v2.json), Offline (82), and four by the services their tasks need. Tasks 103 and 104 are not supported: their evaluate() needs FreeCAD on the driver's machine. Review and the run's start ask for Docker and for the scoring programs (ImageMagick, ffmpeg, tesseract) on the driver's machine. As for OSWorld1, the Where step greys out the container card under the Original, whose docker provider needs Docker on the machine. A unit that ends rate-limited is not run again by itself (limits.run_again: false): an episode is long and paid. A guide,docs/benchmarks/osworld2-machine.md, prepares a machine, runs the null control on one task as its last setup step, and says what a run takes in time and money.Touches
catalogue · templates and static (the mark) · docs and site (
docs/benchmarks/, the generated pages,AGENTS.md) · routes (tests only) · packaging (the entry's.yml,.Dockerfileandstartup/*.pyas package data inpyproject.toml)Before and after
The Benchmarks page before this change
OSWorld2 listed, with its mark
The builder's Benchmark step under Computer use: OSWorld1, beside the selected GAIA
OSWorld1 and OSWorld2
The Agents page: "n/7 benchmarks it runs"
"n/8": OSWorld2 counts, and no agent runs it
Settings, Custom keys: "No custom keys yet."
OSWorld2's
GITLAB_PRIVATE_TOKEN, not setAn agent's page, Codex: the benchmarks it runs
OSWorld2 listed: it needs
agent.computer/v1, which Codex does not offerOSWorld2's page: its setup and its proofs by dataset
Its options: the agent, steps, thinking, the VM provider and its services
Its dataset and its Good to know line
The first of its limitations
Dataset step: six datasets with their controls, and GitLab's token not set
Agent step: OSWorld-V2 agents; the Agents page's agents lack computer use; the settings every agent shares
The Original's own settings: M3 or Muse Spark, steps, thinking
Muse Spark chosen: Thinking, an M3 setting, is gone
Model step, Like the paper: MiniMax-M3 on OpenRouter, Claude Sonnet 4.6 as the judge
Where step: no container backend here; Docker and the scoring programs checked on this machine
Review: every decision, both prerequisites ready, no model key in this fresh profile
The null control's merged job on test_v2, its first rows
052 by the test stand's model in M3: 1.0
Taken at 1400 px, light theme, on a fresh profile. Cut to part of their page: the Benchmark step, Agents and Settings pairs (the top 1000 px), Codex's page (the top 1100 px), OSWorld2's page (four parts) and the null control's job (its first rows). The two jobs are runs named under Verified, opened in the app. The Settings pair, the Model step and Review were taken again after main's Settings icons and Hosts (#163). After the rebase onto main at
5bcfd82d, Settings and the builder's steps were taken on main and on this branch and compared with them: only the Model step changed, where #124 as merged shows Served by as one row, and it was taken again at592be82f.Verified
592be82f, on feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control #133 (a6d32e74) on main at5bcfd82d, on Python 3.12: backend suite (python3 app/tests/parallel.py), 2561 tests in 141 modules, OK; route suite with Node on PATH, 491 tests in 44 modules, OK; the wheel's check (app/packaging/verify.py), OK.test_standard_gated.py, its tests in the classDefinitionoftest_standard_relay_pin.py)./ui-sweep --only osworld2atfc0bdb1f, this pull request's third commit before the rebase (573e8d09now, the entry's files unchanged): 6 screenshots, none with a page error or cut text.PYTHONPATH=app python3 -m nullius standard check: ok.python3 app/tests/catalogue_docs.py --write, then the check: up to date.python3 app/tests/field_contract.py: ok.app/site/readme_media.py --check: current. The docs check reports the same two findings inAGENTS.mdas on main.c3637ffdis this pull request's tree with main's two later commits merged in;f74d1a00and58c2b586add one test commit: two JSON files, no code. It gives OSWorld2's Original one more model, MiniMax M3.1 Flash, the name it is served as, its pin to MiniMax and--max_tokens 32768. That model is a test stand, not in this pull request;58c2b586is this entry on the core before feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124;34e35370and8e2e381bare of the port before its move to this repository's history.standard.json'sproven:fc0bdb1f, this pull request's third commit before the rebase (573e8d09now); the fourth only writes this record, and the fifth changes three tests. The null control ran on all 108 tasks, one launch with each task a shard of its own, at 16 MB/s and 160 IOPS a VM, in six lanes with at most seven VMs at a time; it took 1 h 32 min. 108 and 093 failed their setup under that load and passed alone at 24 MB/s and 240 IOPS, in 24 more minutes. The grader cost $0.14.nullius standard prove osworld2 --dataset test_v2 --job nop=<merged job>passed: the 108 ids are the same twice, 0 of 108 at full score, a mean of 0.0021 against a bound of at most 0.01. Five tasks score above 0 on the untouched VM, as a limitation says;34e35370, which had the same mean; 077's LLM grader gave 0.054 both times.Seven kinds, on
58c2b586, by the test stand's model in OSWorld-V2's M3 agent, from the command line:Each of the seven gives its recorded score again at
c3637ffdwhen its outcome is computed from its saved records, with no VM and no evaluate().The eighth, the simulated user: 007, 0.8, by Muse Spark on
8e2e381b; the simulated user answered 7 times. It was not run on this base.c3637ffd, the null control on 087, a graded task: 0.0. The grader's call went to Anthropic by the definition's pin, andstandard.servedreadsheld: true;f74d1a00, the test stand's model on 052: 1.0. All 68 calls went to MiniMax by the test commit's pin and named it,held: true, with no error.90fde73e), and scored 0.0: the agent declared the task failed at its 120th step. Muse Spark ran none. Their scores before the move, on34e35370and earlier:Risk
agent.computer/v1. Settings shows OSWorld2's GitLab token under Custom keys to every user.osworld-v2.1, the grader through OpenRouter, and by default 100 steps with thinking off. A wrong error kind would move a unit between "environment error" (left out of the adjusted rate) and the agent's error (counted).-jbounds them.startup/sitecustomize.pywraps the creation of the VM's container: its network, its ports on 127.0.0.1, its disk and its write limits. The VMs need a free Docker subnet.