Repository navigation
feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control - #133
Conversation
Closes #136 ## What On Runs → By launch, an agent or control with 2 or more shards is now one row, when `cli merge` would merge those shards. The row shows the numbers of the merged job. `launches.fold` makes that job in memory: the trials of every shard, and the job record from `merge.result_of`. `result_of` is the code that `merge` used inline to compute the job record, now a shared function. `merge.check` keeps the trial records that it already reads. - The rate is over every trial with a verdict, never an average of shard rates. - Units that no shard ran come from `merge.check` and stand beside the count ("2 of 8 not run"). They are never misses. The notes of `merge` are the tooltip. - The state is the most telling state of the shards, with a warning such as "1 of 2 shards stopped". - The header gives the dataset size, not the size of one shard. The row is folded by default. A click on the row, or Enter or Space on its chevron, unfolds one row per shard ("shard 2 · host-2"), each with its Open. The fold uses the shared chevron and slide of `disclosures.js`, and respects reduced motion. A refresh of the list keeps an unfolded row unfolded. Controls fold the same way, stay unranked and stay out of Compare. The task-check line is judged over every shard: "Task checks incomplete: oracle not run on shard 2." A participant with one shard keeps its plain row. Shards that `merge` refuses also keep their rows. Job pages, `cli merge` and merged jobs do not change. The header counts every shard ("8 runs"), every shard has its row, and each folded row says how many shards it holds ("2 shards"). A fold reads every trial record of its shards. It is kept in memory for the shard jobs that the workspace cache gives, and it is read again only when the workspace parses a shard again. A new set of shards replaces each kept fold that shares a shard with it, so the cache holds one fold per shard at most. ## Touches backend · the standard (`merge.py`) · routes · templates and static · docs and site ## Before and after One launch over 2 machines, 8 units: 2 agents, nop and oracle. Shard 2 stopped while terminus-2 ran (2 of its 4 units), and the oracle did not run on shard 2. Light theme, 1280 px. | Before | After, folded | | --- | --- | | <img width="980" height="670" alt="1-before-shards-apart" src="https://github.com/user-attachments/assets/990ceda8-8212-4003-ab1c-f380be783fc9" /> | <img width="980" height="421" alt="2-after-folded" src="https://github.com/user-attachments/assets/f37d9af1-7b3b-4b05-abc6-7dca2da1c983" /> | Before: 8 rows, 2 rates for each agent, "4 tasks", "oracle not run". After: 4 rows, "8 tasks". terminus-2 shows 50% and "3/6 · 2 of 8 not run" with "1 of 2 shards stopped". The task-check line reads "Task checks incomplete: oracle not run on shard 2." | After, unfolded (click and keyboard) | After, dark, 390 px | | --- | --- | | <img width="980" height="595" alt="3-after-unfolded" src="https://github.com/user-attachments/assets/b30f3189-d737-4658-8701-a3bbc434ad4a" /> | <img width="358" height="844" alt="4-after-dark-390" src="https://github.com/user-attachments/assets/6c4b106c-218b-4308-a934-ecac1f96934d" /> | ## Verified - Backend test `Fold.test_a_fold_reads_as_the_merged_job`: three shards, one absent, and one with a planned unit that did not run. The folded job equals `parse_job` of the `merge.merge` output on every field the row shows (solved, rates, outcomes, planned, errors, start, end, time, tokens, cost, mean score, budget). The rate is not the mean of the shard rates. The units not run and the notes are those of `merge`. Shards that `merge` refuses are not folded. - Backend test `Fold.test_a_fold_is_read_again_only_when_the_workspace_parses_a_shard_again`: a second fold of the same cached jobs does not call `merge.check`. After a trial record changes, it calls `merge.check` again. After a third shard appears, the cache holds only the fold of the three. - Route tests `test_launch_shards`: 8 shards give "8 runs", 8 shard rows and 4 folded rows that each say "2 shards". The rows are folded by default. Each shard row has its Open, and the folded row has none. The rate and time of the folded row equal those of the merged job. "2 of 8 not run" and "1 of 2 shards stopped" show. The controls fold and stay out of Compare. A nop that passes one task on shard 2 gives "A control did not behave". A stopped nop shard gives "nop stopped on shard 2". Participants with one shard keep their plain rows. - On `c7af06d5`: backend suite `python3 app/tests/parallel.py` (1989 tests) and route suite `python3 app/tests/parallel.py app/ui_tests` (328 tests) pass in a clean container. - Browser, with the fixture above: folded and unfolded, light and dark, 1280 and 390 px. No page errors and no horizontal page scroll. A click on the row unfolds it, and the shard rows slide open and closed. Tab moves the focus to the chevron of the oracle, Enter unfolds it, and Space folds it again. With reduced motion, the rows show at once. At 390 px, a long agent name wraps and the chevron stays on the line of the name. The name, its "2 shards" line and the shard rows start at the same left edge. - Render time of `/runs?view=launch`, median of 15 requests after a first one, with the same parsed jobs given to the page. 2 agents and nop and oracle, each 4 shards of 500 units (8000 trials): 71 ms before this change, 101 ms after. The first render reads the trial records once: 540 ms, against 199 ms before. The rest of the added time is the outcome bars of the 4 folded rows, over 2000 trials each. 8 shards of 1 unit each (32 trials): 11 ms before, 13 ms after. ## Risk If the fold is wrong, a row shows numbers that differ from the merged job. The backend test compares the two directly. `merge` writes the same `result.json` as before through `result_of`, and its tests pass unchanged. The first render of a large sharded launch reads all its trial records once (+0.3 s for 8000 trials). Later renders use the cache until a shard changes. The cache holds one fold per shard at most, with the parsed jobs that the workspace cache also holds. Open pull requests #124, #133 and #134 also change `merge.py`: the second to merge must rebase. Revert the commit to undo.
…l everywhere (#148) Closes #147 ## What Each model provider is now one record that every feature reads. A feature asks for the protocol that it speaks, and gets the providers that serve it. - **The registry** (`app/nullius/providers.py`). `PROTOCOLS` defines each wire format once: its default path and how the key is sent (`chat`, `responses`, `messages`, `systemone`, `decisions`, `openai-decisions`). `PRESETS` holds OpenRouter, OpenAI, Anthropic and TypeSafe. `record()` returns `{name, base, key, endpoints}` for a preset or a custom provider. `key` is the name of the secret setting, never the key. - **Storage** (`app/nullius/settings.py`). A record is one JSON setting of a new kind, `provider`. A custom provider is `MODEL_<ID>`. The endpoints added to a preset are `OPENROUTER`, `OPENAI`, `ANTHROPIC` or `TYPESAFE`. For these settings the environment does not replace the file. Its value is a partial object, merged over the file's value (RFC 7396, `null` removes a field). Settings shows read-only only the fields that the environment sets. Keys keep their names and stay separate secrets. - **Migration.** `MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` are still read as `name` and `base`. Saving a card moves them into the record. `nullius migrate --providers` previews that rewrite of `settings.json`, and `--apply` writes it. Recipes do not change. `local` and `local:<ID>` still read as `custom`. - **Consumers.** The registry is read by the builder, the provider test, onboarding, `execution`, standard runs, the relay, the CLI's `--provider` and both judges. The builder's JavaScript copy of the list is gone. The builder offers a provider only where it serves what the recipe's agents and judge speak, so Anthropic appears for an agent of the standard that speaks `messages`. Review, Run (`web/execute.py`) and `run.Run` refuse a provider that does not fit. The relay sends `x-api-key` and `anthropic-version` on a `messages` route only, and a bearer header on a benchmark's own routes. - **Settings cards.** A card shows Base and a collapsed Endpoints section with one row for each protocol. The placeholder is the default URL. A base on a preset's host suggests that preset's endpoints. A preset's known endpoints are fixed. An endpoint added to a preset is a path, or a full URL on the preset's own host, because the preset's key goes there. - **System One** (`app/nullius/judge/client.py`) asks each decision protocol: the TypeSafe shape on `systemone` and `decisions`, and its own adapter for `openai-decisions`. It keeps the provider's `usage.cost` and records a `refusal` as no answer. A 402 says what it means. A judge never follows a redirect with the key. - **Judge cards.** Each judge has Provider, then Protocol when the provider serves two protocols of that judge (else one quiet line, "via ..."), then Model. Each judge's provider is a select whatever the count, because the list is the user's own providers and grows or shrinks with them; cards are kept for fixed sets of choices. New settings: `LLM_JUDGE_PROTOCOL`, `SYSTEM_ONE_PROTOCOL`. On OpenRouter, System One offers OpenRouter's public list of decision models, asked without a key, cached 6 hours, with the fixed list as the fallback. The seven commits are atomic and in reading order: registry and storage, consumers, Settings cards, decision protocols, judge cards, tests, docs. ## Touches backend (`providers`, `settings`, `migrate`, `onboarding`, `execution`, `judge/client`, `recipes/launch`) · the standard (`run`, `relay`, `recipe`, `cli`, `definitions`, `fidelity`) · routes (`web/settings`, `web/execute`, `web/filters`) · templates and static (Settings cards, judge cards, builder core) · docs (`docs/settings/model-providers.md`, `system-one.md`, `llm-judge.md`, `README.md`, `agent-keys.md`, `docs/builder.md`, `docs/judgement.md`, `docs/standard.md`, `docs/cli/README.md`, `AGENTS.md`, `app/.env.example`) ## Before and after Settings at 1280 px, light theme. Fixture keys only. Model providers. TypeSafe is a preset like the others, so it comes after the run providers. | Before | After | | --- | --- | | <img width="748" height="431" alt="1-before-providers" src="https://github.com/user-attachments/assets/1dd2338e-0b32-4631-8895-a6df2ba7fc29" /> | <img width="748" height="431" alt="2-after-providers" src="https://github.com/user-attachments/assets/ff3bb210-22ef-4bff-bbe4-df4ac4488519" /> | A preset card (OpenRouter), Endpoints open. Known endpoints are fixed. Other protocols are editable rows. | Before | After | | --- | --- | | <img width="748" height="360" alt="3-before-preset-card" src="https://github.com/user-attachments/assets/5fb776e9-1c86-4cf0-8508-24041b6f9042" /> | <img width="748" height="706" alt="4-after-preset-card" src="https://github.com/user-attachments/assets/9058e8e4-339e-49da-8a44-67b2ffcdf8bf" /> | A custom provider, Endpoints open, with one endpoint set. | Before | After | | --- | --- | | <img width="748" height="451" alt="5-before-custom-card" src="https://github.com/user-attachments/assets/c725f642-3d22-4b8d-8fb0-c702e415594d" /> | <img width="748" height="897" alt="6-after-custom-card" src="https://github.com/user-attachments/assets/afcbcb01-fed9-4ef3-87c2-883d73594edc" /> | Judges. System One on OpenRouter serves two of its protocols, so it shows Protocol. The LLM judge on OpenAI serves one, so it shows "via OpenAI Chat Completions". | Before | After | | --- | --- | | <img width="748" height="504" alt="7-before-judges" src="https://github.com/user-attachments/assets/a089a1bd-9f72-4ca6-bff1-e1a652a83ec1" /> | <img width="748" height="503" alt="8-after-judges" src="https://github.com/user-attachments/assets/61ef9ae3-e080-4e54-99a1-dc43d4960ba6" /> | The custom card at 390 px: <img width="358" height="1153" alt="9-after-custom-card-390" src="https://github.com/user-attachments/assets/dce35d02-44b1-4ccb-8362-c84129e1eb87" /> ## Verified The branch is based on `main` at 1cc37ac. I tested its tip, each suite in a new rootless Podman container (`python:3.12`, `--init`, uv on PATH, Node for the route suite): - Backend suite (`python3 app/tests/parallel.py`): 2029 tests in 118 modules, OK. `main` at 1cc37ac, in the same container: 1992 tests in 116 modules, OK. - Route suite (`python3 app/tests/parallel.py app/ui_tests`): 441 tests in 42 modules, OK. `main` at 1cc37ac: 431 tests in 41 modules, OK. - New modules: `test_provider_registry` (17 tests: the merge, `null`, partial environment values, the old settings, record checks, the preset host rule, keys never in a record), `test_judge_decisions` (18 tests: each decision protocol's request and reply, `usage.cost`, `refusal`, 402, no redirect, the cached model list and its fallback), `ui_tests.test_judge_cards` (2 tests: the protocol row and the live list). Existing modules gained tests for the relay's headers, the builder's provider filter, Harbor recipes on Anthropic and the Settings cards. - End to end, against the server (`app/app.py`) and the CLI, in a container with no network, with a new `NULLIUS_HOME` for each case and fixture keys only. 37 checks, all pass: - No settings: the four presets and the OpenAI-compatible card. - A custom provider in `settings.json`: its card, its merged record, the default path for a protocol without an entry, System One offers it for its `decisions` endpoint, no key in the page. - `MODEL_GW='{"base": "https://gw-eu.example/v1", "endpoints": {"decisions": null}}'` in the environment: the base changes, the name stays the file's, the endpoint goes, System One no longer offers it. The card shows Base read-only and Name editable. The file does not change. - `nullius migrate --providers`: the preview names the moves, prints no value and leaves the file as it was. `--apply` writes the record, removes the old lines, keeps the key and the file mode 0600. A second run finds nothing to move. - Old variables in the environment (`MODEL_API_URL`, `MODEL_<ID>_NAME`, `MODEL_<ID>_API_URL`, `LLM_JUDGE_PROVIDER=local:<ID>`): read as before. - System One's model list on OpenRouter, offline: the route answers with the built-in list in less than the 5 s timeout, or with a cached list, however old. - Browser (Chromium, Playwright), on this branch and on `main`, at 1280 px and 390 px, light and dark, with no network: the provider list, a preset card and a custom card with Endpoints open, and both judge cards. No page errors and no horizontal scroll. Not verified: live calls to OpenRouter Decisions and OpenAI Decisions, and a whole run on Anthropic. The tests use fixtures. ## Risk A wrong record or header can send a call to the wrong URL, or with the wrong key header. Undo with a revert. The old settings are still read, so a revert keeps working with them. Records written by a card stay in `settings.json` and the older code ignores them. - **Migration on edit.** Saving a custom provider's card moves `MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` into `MODEL_<ID>`. After a revert, that provider loses its name and base until they are set again. `nullius migrate --providers` without `--apply` shows what would move. - **New reserved names.** `OPENROUTER`, `OPENAI`, `ANTHROPIC`, `TYPESAFE` and `MODEL_<ID>` are now settings. An agent key listed under one of these names is left out, and Settings says so. A shell variable such as `MODEL_PATH` that is not a JSON object is not read as a record. - **`--provider anthropic` on the CLI.** The key header now follows the agent's protocol. An agent that speaks only `chat` is refused on Anthropic, where it used to get `x-api-key` on OpenAI-style paths. - **Runs and custom endpoint paths.** The relay forwards the agent's own path below the provider's base. An Endpoints row changes where the judges and the test ask, not where a run's calls go. - **`openai-decisions` field names** come from OpenAI's documentation, not from a live reply. - **Open PRs on the same files.** This change does not depend on any of them. Whichever merges second rebases. A test merge of this branch with each one: - #124 (OpenRouter serving providers, `--served-by`): conflicts in `providers.py`, `standard/run.py`, `standard/relay.py` and `docs/settings/model-providers.md`. Each conflict is small. Both add functions at the same place in `providers.py`. This branch removes `run.PROVIDERS`, where #124 adds `routing` to the OpenRouter entry: that flag becomes a field of the OpenRouter preset. Both change `Relay._headers`. - #133 (stacked on #124): the same four files, and also `standard/fidelity.py` (one `key_header=` line, now `r.key_header`) and `docs/standard.md` (one table row each). - #134 (stacked on #133): the same as #133, and also `ui_tests/test_settings_page.py` (one placeholder assertion, which both change). - #124, #133 and #134 also conflict with `main` alone in `standard/merge.py` (and #133, #134 in `tests/test_standard_shard.py`), which this branch does not touch.
…on top of --served-by: gated tasks, a definition's pins as its serving pins, a pinned route's refusal as the unit's, limits of their own, and prove of a merged control A definition's pin (setups.original.pin, score.judge.pin) joins the run's --served-by pins and wins over --served-by for its model; ApiProviderRouteError reads the pin and reply.served_by. A failed Responses object reads reply.end and reply.error.
…eep the score of a unit a guard stopped despite a last 429 limits.run_again false: a unit that ended rate-limited is not run once more; its outcome stands. A unit whose driver scored it after a guard's stop keeps that score, marked stopped, though its last own call got a 429.
… own last 429, not a side call's or a context window's stop The rule reads the agent's own last call, a guard's line in the stop file and a score stopped by other than the context window; a cap's refusal is still the cap's. limits.run_again takes null as unset. The rate limits row names the opt-out.
d19a802 to
a6d32e7
Compare
| raise Refused(f"--served-by names {', '.join(stray)}, which this run does not call: it calls " | ||
| + (", ".join(called) or "no model")) | ||
| self.route_pins = pins_of(bench, other, provider, self.judge_model) | ||
| self.serving.update(self.route_pins) |
There was a problem hiding this comment.
The definition's pins override Served by here, but the builder never shows them.
self.serving.update(self.route_pins) makes setups.original.pin and score.judge.pin win over the user's --served-by for the same model. That is the intent. However, the builder's Served by (#124: command-builder.js, recipe.py served_by) does not know these pins exist. It still says "any provider" and offers every provider for a pinned model, so the page offers a choice the run then ignores.
Reproduced with OSWorld2 (#134, which pins MiniMax-M3 to minimax and the official judge to anthropic): choose Like the paper, OpenRouter, MiniMax-M3. Served by lists 8 providers (CoreWeave, DeepInfra, GMICloud, Minimax, Parasail, ...).
| Served by in the builder | What the run sends |
|---|---|
| empty ("any provider") | minimax only |
| DeepInfra | minimax only, with no notice |
| the judge's Served by, anything (official judge) | anthropic only |
Review does not say the user's pin will be replaced.
Suggested fix:
- the recipe compile (which already returns
served_models) also returns each role's definition pin, computed by the same rule aspins_of; - the builder draws a pinned role's Served by as a fixed value (the paper's providers), with no menu;
- the recipe check, and
--served-byon the CLI, refuse a pin that disagrees with the definition's for that model, instead of overriding it silently.
This could land here or in #134. #134 is where it first becomes visible, since OSWorld2 is the first entry with a pin.
There was a problem hiding this comment.
This also clashes with the design of #124, which added --served-by. There, the serving provider is the user's choice: nothing is pinned unless the user asks, and an empty Served by means any provider. This PR has the definition pin some models whatever the user picks. Those are two different rules for who decides the provider.
Before choosing a fix, the authors of #124 (pierreleca) and this PR (sirotaoleg) should agree on one rule. For example:
- the definition's pin is only a default, shown in Served by, that the user can change; or
- the definition's pin is fixed for the Original and the official judge, and Served by shows it as fixed for those models.
… work on a unit and cap its model calls (#191) Closes #190 ## What A standard run can now force its limits, and the agent's time limit covers its whole work on a unit. - **Time limits by phase.** `run --timeout PHASE=SECONDS` forces the limit of a phase: `setup`, `image`, `agent`, `verify` or `score`. Empty keeps the default, which comes from the task, then the benchmark, then Nullius. `config.json` records what was forced under `standard.timeouts`. - **One agent limit per unit.** The agent's limit is one deadline for its whole work on a unit: every turn of a conversation, and the simulated user's time between them. Before, it applied to each reply. A capability keeps one limit per add and per recall, because a memory adds a unit's sessions one by one. τ³-bench's `limits.turn_seconds` goes from 600 s a reply to 7200 s a conversation. - **Turns.** `run --max-turns N` sets the turns of a conversation. The builder offers it only on a benchmark that holds conversations. - **A cap on model calls.** `run --max-calls N` caps the model calls an agent makes on a unit. The relay counts each answered call on the agent's route; rate-limited and failed calls are not counted, and neither are the judge's or the simulator's. Past the cap the relay refuses the call with a 402, and the unit ends as `UnitCallCapError`, an agent error that is not scored. It is for agents only: a capability's route also carries the benchmark's reader, so a capability run refuses it. - **A benchmark's own bounds.** They are now options marked `limit`. τ³-bench gets Steps per conversation (`--max-steps`, from 1 up to its default of 200) and Tool errors per conversation (`--max-errors`). OSWorld gets Steps per task (each agent script's `--max_steps`); left empty it uses the chosen agent's own value, declared with the new `by` key. - **Builder, Limits step.** It shows Scope, then Time limits in minutes, then a section named after the benchmark when it has bounds of its own. An empty field shows the value the run uses: in the text colour when the benchmark or the agent declares it, grey when it is Nullius's default (priority: Nullius < agent < benchmark). Nothing is typed into a field for the user. ## Touches - `app/nullius/standard/`: `run.py`, `relay.py`, `container.py`, `job.py`, `cli.py`, `recipe.py`, `definitions.py` - `app/nullius/runs/classify.py`: `UnitCallCapError` is an agent error - Catalogue: τ³-bench and OSWorld's `standard.json`, their drivers and `upstream-settings.json` - `app/static/builder/`: the Limits step - Docs: `docs/cli/run.md`, `docs/standard.md`, `docs/builder.md`, the field guide and the generated supported-benchmark pages ## Before and after The Limits step for τ³-bench, before and after: | Before | After | | --- | --- | | <img width="440" alt="Limits step on main: Scope only, with tasks, concurrency, attempts and two spending caps" src="https://github.com/user-attachments/assets/80b468a4-326f-4057-a4b2-5551ea93e642" /> | <img width="440" alt="Limits step on this branch: Scope with turns and model calls, Time limits in minutes, then a tau3-bench section with steps and tool errors" src="https://github.com/user-attachments/assets/6278494c-6c9f-456f-b739-9578494315ee" /> | ## Verified - Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging `main`, with #124's serving providers and #150's Boat VMs). - New tests: - A whole conversation that ends past the agent limit, though no single turn does. - Four one-call turns cut on the fourth call by `max_calls=3`. - The relay counts only answered calls on the agent's route. - A slow capability whose three adds each fit a 1 s limit while their total does not. - A capability run refuses `--max-calls`. - Recipe validation of the limits, including a value that rounds to 0 s. - τ³-bench steps above 200 are refused. - OSWorld's `--max_steps` is replaced, or added when the agent's command doesn't set it. - Against the installed τ³-bench upstream: the driver uses 200 steps and 10 errors by default, and 7 and 3 when the run sets them. - Checked in the browser, light and dark, for τ³-bench and OSWorld. ## Risk - **Agent timeouts on conversation benchmarks.** A conversation that used to fit 600 s per reply now has a single limit for the whole conversation, 7200 s on τ³-bench. A very long conversation can now time out where it didn't before. Undo by forcing a larger `--timeout agent=SECONDS`, or by reverting `limits.turn_seconds`. - **Comparability.** A run whose limits differ from the defaults is not yet marked as not comparable with the leaderboard. - **Overlap.** #174 (the agent's setup timed apart from its run) and #133 (a benchmark's own limits) change the same parts of `run.py` and the catalogue, so expect merge conflicts.
What
A benchmark of the standard can now say more about how its units run and end, which a computer-use benchmark like OSWorld2 (#134) needs. The pins,
gated, the new limits andprove's new options are opt-in. The relay's rules and the rules of how a unit ends change for every benchmark of the standard; Risk lists them.Gated tasks (
gated: true): a unit's failure that the relay's records explain, a model endpoint's (the agent's, the judge's or the simulated user's), says in itsresult.jsonthe status and class alone, never the provider's words, which can quote the request. The trial's call records keep the provider's words, and an error a driver raises itself is kept as the driver words it.A definition's providers (
setups.original.pinfor the Original's models,score.judge.pinfor the official judge; OpenRouter only) join the run's--served-bypins (feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124) and win over--served-byfor the same model: a run of the Original and its official judge always go to the providers the paper used. The pin follows the model sent, so a model the agent shares with the judge goes to the judge's providers. A refused route fails the unit withApiProviderRouteError, an API error outside the agent's rate, not a miss:reply.served_byshares nothing with the pin's slugs. A name that is not its slug (Googleforgoogle-vertex) fails nothing on a guess.The agent's own last call's error comes first, and a setup error stays a setup error. Three units in a row whose pinned route has no endpoint stop the run, as a refused key does. A user's
--served-byon any other model behaves as in feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124.Limits of a benchmark's own:
limits.score_seconds: how longscoremay take;limits.call_seconds: how long the relay waits for one model call, alsorelay serve --timeout;limits.run_again: false: a unit that ended rate-limited is not run once more, and its outcome stands;limits.turn_seconds, read before, is now documented.How a unit ends:
EnvironmentSetupError:,VerifierError:,CancelledError:orCauseUnknownError:keeps that kind;ContextWindowExceededError, the agent's;scorecan say the stop's reason (verifier_result.stopped);OutputLengthExceededErroronly when its last reply has neither text nor a tool call. When the answerer fails and its last reply was a Responses object answered 200 andfailed, that is an API error.The relay:
answered: 200). feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 leaves counting an error inside a 200 to a pull request of its own; this one takes only a 200 with no reply in it;stopfile beside the relay's records: the relay notes in it an agent's call refused for the model's context window, and that note refuses nothing. Once a benchmark's guard writes a line of its own there, the agent's later calls are refused unsent, while side models and the judge's and simulated user's routes go on;A job's name:
.work/relay,result.json,verifier/,agent/) aside to.work/earlier.<ns>.prove and merge:
nullius standard prove ... --bench-option NAME=VALUEsets the benchmark's own options for the units check and the controls;--job nop=MERGED(ororacle=) holds a merged launch of a control to its bound instead of running it, checking every trial's own record;mergeaccepts shards that differ in benchmark options and keeps each shard's inmerged_from;mergerefuses, unless--force, a trial whose record names another job, benchmark, version, control or agent, and a job itself merged despite differences. It lists the shards absent from a merge only when the launch has at most 1,000.Smaller:
agent/trace.error.txt;thinking_tokenscount as reasoning;comparablecan list the models whose runs compare with the paper, and the builder's message for another model says so;Read first:
failure,route_failureandroutes_failure, then_hollowand the stop inRelay._forward(app/nullius/standard/relay.py). Next,pins_of,route_pinsand the outcome rules inRun(app/nullius/standard/run.py), then the--jobcheck inprove.pyand the trial check inmerge.py. Thenapp/tests/test_standard_relay_pin.py(classesPinnedandWired),test_standard_gated.py,test_standard_context_window.pyandtest_standard_controls.py. The rest is the definitions' checks, the CLI options, the sidecar's argument and the pages.Touches
the standard (
app/nullius/standard/:relay,run,prove,definitions,merge,job,cli,container,recipe,fidelity) · docs and site (docs/standard.md,docs/cli/run.md,docs/cli/read-jobs.md)Verified
a6d32e74, on main at5bcfd82d, on Python 3.12: backend suite (python3 app/tests/parallel.py), 2186 tests in 126 modules, OK; route suite with Node on PATH, 491 tests in 44 modules, OK; the wheel's check (app/packaging/verify.py), OK.5bcfd82d): 10 fail, 2 pass, and 1 does not end.test_standard_relay_pin: all 15;test_standard_context_window: 19 of 20;test_standard_controls: 17 of 43;test_standard_relay: 8 of 58;test_standard_gated: 4 of 6.test_standard_run: 2 of 50;test_standard_trace: 2 of 20;test_standard_parts,test_standard_served_byandtest_standard_units: 1 each.test_standard_sharddoes not end within 4 minutes: one of its merges names 2^64 shards, and main'smergelists every absent one.test_standard_versionsandtest_dependency_surfacepass on both.WiredorPinnedfail:--served-bypin;setups.original.pinis checked and reaches the run, and thatlimits.call_secondsreaches the relay and the sidecar's--timeout. OSWorld2 is the first entry to set them (test_osworld2_muse.py, and the tests it adds to the classDefinitionoftest_standard_relay_pin.py).test_standard_served_bypasses as main has it;write_shardintest_standard_shardtakes feat(runs): fold a launch's shards into one row per participant #137'srewardbeside this branch'sbench_options.PYTHONPATH=app python3 -m nullius standard check: ok.python3 app/tests/catalogue_docs.py --write, then the check: up to date.python3 app/tests/field_contract.py: ok.app/site/readme_media.py --check: current. The docs check reports the same two findings inAGENTS.mdas on main.c3637ffdis this code with main's two later commits merged in and OSWorld2 on top, a commit of the port's working branch:c3637ffd, the null control on one task with an LLM grader: the grader's call went out withrequest.routing{"order": ["anthropic"], "allow_fallbacks": false}from the definition's pin,reply.served_bynamed Anthropic, andstandard.servedreadsheld: true;c3637ffdplus a test commit that adds one more model to OSWorld2's Original with its pin to MiniMax, an agent episode: all 68 calls named their provider,held: true, no error.--timeout; the docker tier; the site build.Risk
ContextWindowExceededError), where it was an API error outside its rate;rate_limitedand runs it once more, where its score stood;failedResponses object is an API error, where it was the agent's;proveruns no control when the units check failed, and fails a control job with a unit that has neither a score nor an error;mergerefuses a trial whose record names another job unless--force.5bcfd82d), and these three commits sit on it, rebased: its--served-bycode is main's as merged, and the pins are added on top.