Skip to content

feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control - #133

Open
sirotaoleg wants to merge 3 commits into
ThinkFlowLab:mainfrom
sirotaoleg:feat/benchmark-pins-limits
Open

sirotaoleg wants to merge 3 commits into
ThinkFlowLab:mainfrom
sirotaoleg:feat/benchmark-pins-limits

Conversation

@sirotaoleg

@sirotaoleg sirotaoleg commented Nov 15, 2026 •

Copy link
Copy Markdown

What

A benchmark of the standard can now say more about how its units run and end, which a computer-use benchmark like OSWorld2 (#134) needs. The pins, gated, the new limits and prove's new options are opt-in. The relay's rules and the rules of how a unit ends change for every benchmark of the standard; Risk lists them.

  • Gated tasks (gated: true): a unit's failure that the relay's records explain, a model endpoint's (the agent's, the judge's or the simulated user's), says in its result.json the status and class alone, never the provider's words, which can quote the request. The trial's call records keep the provider's words, and an error a driver raises itself is kept as the driver words it.

  • A definition's providers (setups.original.pin for the Original's models, score.judge.pin for the official judge; OpenRouter only) join the run's --served-by pins (feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124) and win over --served-by for the same model: a run of the Original and its official judge always go to the providers the paper used. The pin follows the model sent, so a model the agent shares with the judge goes to the judge's providers. A refused route fails the unit with ApiProviderRouteError, an API error outside the agent's rate, not a miss:

    • a pinned call answered 404 or 503 when it is that model's last call on its route;
    • any pinned call whose reply.served_by shares nothing with the pin's slugs. A name that is not its slug (Google for google-vertex) fails nothing on a guess.

    The agent's own last call's error comes first, and a setup error stays a setup error. Three units in a row whose pinned route has no endpoint stop the run, as a refused key does. A user's --served-by on any other model behaves as in feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124.

  • Limits of a benchmark's own:

    • limits.score_seconds: how long score may take;
    • limits.call_seconds: how long the relay waits for one model call, also relay serve --timeout;
    • limits.run_again: false: a unit that ended rate-limited is not run once more, and its outcome stands;
    • limits.turn_seconds, read before, is now documented.
  • How a unit ends:

    • a driver's error that begins EnvironmentSetupError: , VerifierError: , CancelledError: or CauseUnknownError: keeps that kind;
    • the agent's own last call refused as longer than the model's context window is ContextWindowExceededError, the agent's;
    • a last 429 is read from the agent's own calls in the order they began, so a side model's later 200 does not cover it;
    • a unit a guard's stop ended keeps the score its driver gave, its own last call's 429 notwithstanding. score can say the stop's reason (verifier_result.stopped);
    • for an upstream's own answerer: OutputLengthExceededError only when its last reply has neither text nor a tool call. When the answerer fails and its last reply was a Responses object answered 200 and failed, that is an API error.
  • The relay:

    • a non-streamed 200 to a model call with no reply in it reaches the caller and the record as a 502, or the 429 or 402 of an error it held (the record keeps answered: 200). feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 leaves counting an error inside a 200 to a pull request of its own; this one takes only a 200 with no reply in it;
    • a stream cut off midway, after the caller had its 200, is recorded as a 502, and its booking against the spending cap stays whole;
    • a call refused with 400, 413 or 422 records each inline image it sent by its declared type, the type its bytes show and its size, never a byte;
    • a stop file beside the relay's records: the relay notes in it an agent's call refused for the model's context window, and that note refuses nothing. Once a benchmark's guard writes a line of its own there, the agent's later calls are refused unsent, while side models and the judge's and simulated user's routes go on;
    • a failed Responses object with no usage records a cost of 0, and its booking is kept.
  • A job's name:

    • a run holds its job name until its process ends, and one under a name another run still holds is refused before it writes anything;
    • a rerun into the same job moves each rerun trial's earlier records (.work/relay, result.json, verifier/, agent/) aside to .work/earlier.<ns>.
  • prove and merge:

    • nullius standard prove ... --bench-option NAME=VALUE sets the benchmark's own options for the units check and the controls;
    • --job nop=MERGED (or oracle=) holds a merged launch of a control to its bound instead of running it, checking every trial's own record;
    • merge accepts shards that differ in benchmark options and keeps each shard's in merged_from;
    • merge refuses, unless --force, a trial whose record names another job, benchmark, version, control or agent, and a job itself merged despite differences. It lists the shards absent from a merge only when the launch has at most 1,000.
  • Smaller:

    • a trace that cannot be built leaves agent/trace.error.txt;
    • an Original's trajectory keeps its tool calls;
    • each call's record keeps the request's sha256;
    • thinking_tokens count as reasoning;
    • a setup's comparable can list the models whose runs compare with the paper, and the builder's message for another model says so;
    • a test checks that every file a host's snapshot takes of a bundled entry is package data.

Read first: failure, route_failure and routes_failure, then _hollow and the stop in Relay._forward (app/nullius/standard/relay.py). Next, pins_of, route_pins and the outcome rules in Run (app/nullius/standard/run.py), then the --job check in prove.py and the trial check in merge.py. Then app/tests/test_standard_relay_pin.py (classes Pinned and Wired), test_standard_gated.py, test_standard_context_window.py and test_standard_controls.py. The rest is the definitions' checks, the CLI options, the sidecar's argument and the pages.

Touches

the standard (app/nullius/standard/: relay, run, prove, definitions, merge, job, cli, container, recipe, fidelity) · docs and site (docs/standard.md, docs/cli/run.md, docs/cli/read-jobs.md)

Verified

  • At a6d32e74, on main at 5bcfd82d, on Python 3.12: backend suite (python3 app/tests/parallel.py), 2186 tests in 126 modules, OK; route suite with Node on PATH, 491 tests in 44 modules, OK; the wheel's check (app/packaging/verify.py), OK.
  • The 13 changed test modules on main's source (5bcfd82d): 10 fail, 2 pass, and 1 does not end.
    • test_standard_relay_pin: all 15; test_standard_context_window: 19 of 20; test_standard_controls: 17 of 43; test_standard_relay: 8 of 58; test_standard_gated: 4 of 6.
    • test_standard_run: 2 of 50; test_standard_trace: 2 of 20; test_standard_parts, test_standard_served_by and test_standard_units: 1 each.
    • test_standard_shard does not end within 4 minutes: one of its merges names 2^64 shards, and main's merge lists every absent one.
    • test_standard_versions and test_dependency_surface pass on both.
  • Mutations. Each of these changes to the pinned route makes a named test of Wired or Pinned fail:
    • the refusal read against every --served-by pin;
    • the agent's own route read without the definition's pins;
    • the judge's route not read;
    • a route refusal not counted toward the three-in-a-row stop;
    • a provider name taken from a reply unchecked.
  • Covered in feat(catalogue): add OSWorld2 with OSWorld-V2's own agents, its services on the machine and a null control #134, not here: that setups.original.pin is checked and reaches the run, and that limits.call_seconds reaches the relay and the sidecar's --timeout. OSWorld2 is the first entry to set them (test_osworld2_muse.py, and the tests it adds to the class Definition of test_standard_relay_pin.py).
  • feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124's own tests. Three were changed for two deliberate differences: a 200 with no reply is a 502, and a failed Responses object is read as failed. The rest of test_standard_served_by passes as main has it; write_shard in test_standard_shard takes feat(runs): fold a launch's shards into one row per participant #137's reward beside this branch's bench_options.
  • PYTHONPATH=app python3 -m nullius standard check: ok. python3 app/tests/catalogue_docs.py --write, then the check: up to date. python3 app/tests/field_contract.py: ok. app/site/readme_media.py --check: current. The docs check reports the same two findings in AGENTS.md as on main.
  • Against OpenRouter, through OSWorld2 (feat(catalogue): add OSWorld2 with OSWorld-V2's own agents, its services on the machine and a null control #134). c3637ffd is this code with main's two later commits merged in and OSWorld2 on top, a commit of the port's working branch:
    • on c3637ffd, the null control on one task with an LLM grader: the grader's call went out with request.routing {"order": ["anthropic"], "allow_fallbacks": false} from the definition's pin, reply.served_by named Anthropic, and standard.served reads held: true;
    • on c3637ffd plus a test commit that adds one more model to OSWorld2's Original with its pin to MiniMax, an agent episode: all 68 calls named their provider, held: true, no error.
  • Not run: a pinned route refused by OpenRouter in a live run (404 and 503 are covered by the tests); a test of a pinned simulated user's route in a run; a benchmark already on main, live, on this branch; the container backend with --timeout; the docker tier; the site build.

Risk

  • For every benchmark of the standard:
    • a non-streamed 200 with no reply is a 502, and a stream cut off midway is recorded as one;
    • the agent's own last call refused for its context window is the agent's error (ContextWindowExceededError), where it was an API error outside its rate;
    • an agent's last 429 followed by a side model's 200 makes the unit rate_limited and runs it once more, where its score stood;
    • an upstream answerer whose last reply was a failed Responses object is an API error, where it was the agent's;
    • a driver's error of a kind it names: a setup error or a stop is left out of the agent's rate, and a verifier error or one of no known cause counts as unknown. A wrong kind would move a unit between them;
    • prove runs no control when the units check failed, and fails a control job with a unit that has neither a score nor an error;
    • a run under a job name another run still holds is refused, and a rerun into an existing job moves the rerun trials' records aside;
    • merge refuses a trial whose record names another job unless --force.
  • Pins. Only a definition that pins a model is affected; none on main does. For such a model the builder still offers Served by, and the run keeps the definition's providers whatever it says.
  • Records. Stored records gain fields and keep the old ones.
  • feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 is merged (5bcfd82d), and these three commits sit on it, rebased: its --served-by code is main's as merged, and the pins are added on top.
  • Undone by reverting the commits.

harenome added a commit that referenced this pull request Nov 16, 2026
Closes #136

## What

On Runs → By launch, an agent or control with 2 or more shards is now
one row, when `cli merge` would merge those shards. The row shows the
numbers of the merged job. `launches.fold` makes that job in memory: the
trials of every shard, and the job record from `merge.result_of`.
`result_of` is the code that `merge` used inline to compute the job
record, now a shared function. `merge.check` keeps the trial records
that it already reads.

- The rate is over every trial with a verdict, never an average of shard
rates.
- Units that no shard ran come from `merge.check` and stand beside the
count ("2 of 8 not run"). They are never misses. The notes of `merge`
are the tooltip.
- The state is the most telling state of the shards, with a warning such
as "1 of 2 shards stopped".
- The header gives the dataset size, not the size of one shard.

The row is folded by default. A click on the row, or Enter or Space on
its chevron, unfolds one row per shard ("shard 2 · host-2"), each with
its Open. The fold uses the shared chevron and slide of
`disclosures.js`, and respects reduced motion. A refresh of the list
keeps an unfolded row unfolded. Controls fold the same way, stay
unranked and stay out of Compare. The task-check line is judged over
every shard: "Task checks incomplete: oracle not run on shard 2."

A participant with one shard keeps its plain row. Shards that `merge`
refuses also keep their rows. Job pages, `cli merge` and merged jobs do
not change. The header counts every shard ("8 runs"), every shard has
its row, and each folded row says how many shards it holds ("2 shards").

A fold reads every trial record of its shards. It is kept in memory for
the shard jobs that the workspace cache gives, and it is read again only
when the workspace parses a shard again. A new set of shards replaces
each kept fold that shares a shard with it, so the cache holds one fold
per shard at most.

## Touches

backend · the standard (`merge.py`) · routes · templates and static ·
docs and site

## Before and after

One launch over 2 machines, 8 units: 2 agents, nop and oracle. Shard 2
stopped while terminus-2 ran (2 of its 4 units), and the oracle did not
run on shard 2. Light theme, 1280 px.

| Before | After, folded |
| --- | --- |
| <img width="980" height="670" alt="1-before-shards-apart"
src="https://github.com/user-attachments/assets/990ceda8-8212-4003-ab1c-f380be783fc9"
/> | <img width="980" height="421" alt="2-after-folded"
src="https://github.com/user-attachments/assets/f37d9af1-7b3b-4b05-abc6-7dca2da1c983"
/> |

Before: 8 rows, 2 rates for each agent, "4 tasks", "oracle not run".
After: 4 rows, "8 tasks". terminus-2 shows 50% and "3/6 · 2 of 8 not
run" with "1 of 2 shards stopped". The task-check line reads "Task
checks incomplete: oracle not run on shard 2."

| After, unfolded (click and keyboard) | After, dark, 390 px |
| --- | --- |
| <img width="980" height="595" alt="3-after-unfolded"
src="https://github.com/user-attachments/assets/b30f3189-d737-4658-8701-a3bbc434ad4a"
/> | <img width="358" height="844" alt="4-after-dark-390"
src="https://github.com/user-attachments/assets/6c4b106c-218b-4308-a934-ecac1f96934d"
/> |

## Verified

- Backend test `Fold.test_a_fold_reads_as_the_merged_job`: three shards,
one absent, and one with a planned unit that did not run. The folded job
equals `parse_job` of the `merge.merge` output on every field the row
shows (solved, rates, outcomes, planned, errors, start, end, time,
tokens, cost, mean score, budget). The rate is not the mean of the shard
rates. The units not run and the notes are those of `merge`. Shards that
`merge` refuses are not folded.
- Backend test
`Fold.test_a_fold_is_read_again_only_when_the_workspace_parses_a_shard_again`:
a second fold of the same cached jobs does not call `merge.check`. After
a trial record changes, it calls `merge.check` again. After a third
shard appears, the cache holds only the fold of the three.
- Route tests `test_launch_shards`: 8 shards give "8 runs", 8 shard rows
and 4 folded rows that each say "2 shards". The rows are folded by
default. Each shard row has its Open, and the folded row has none. The
rate and time of the folded row equal those of the merged job. "2 of 8
not run" and "1 of 2 shards stopped" show. The controls fold and stay
out of Compare. A nop that passes one task on shard 2 gives "A control
did not behave". A stopped nop shard gives "nop stopped on shard 2".
Participants with one shard keep their plain rows.
- On `c7af06d5`: backend suite `python3 app/tests/parallel.py` (1989
tests) and route suite `python3 app/tests/parallel.py app/ui_tests` (328
tests) pass in a clean container.
- Browser, with the fixture above: folded and unfolded, light and dark,
1280 and 390 px. No page errors and no horizontal page scroll. A click
on the row unfolds it, and the shard rows slide open and closed. Tab
moves the focus to the chevron of the oracle, Enter unfolds it, and
Space folds it again. With reduced motion, the rows show at once. At 390
px, a long agent name wraps and the chevron stays on the line of the
name. The name, its "2 shards" line and the shard rows start at the same
left edge.
- Render time of `/runs?view=launch`, median of 15 requests after a
first one, with the same parsed jobs given to the page. 2 agents and nop
and oracle, each 4 shards of 500 units (8000 trials): 71 ms before this
change, 101 ms after. The first render reads the trial records once: 540
ms, against 199 ms before. The rest of the added time is the outcome
bars of the 4 folded rows, over 2000 trials each. 8 shards of 1 unit
each (32 trials): 11 ms before, 13 ms after.

## Risk

If the fold is wrong, a row shows numbers that differ from the merged
job. The backend test compares the two directly. `merge` writes the same
`result.json` as before through `result_of`, and its tests pass
unchanged. The first render of a large sharded launch reads all its
trial records once (+0.3 s for 8000 trials). Later renders use the cache
until a shard changes. The cache holds one fold per shard at most, with
the parsed jobs that the workspace cache also holds. Open pull requests
#124, #133 and #134 also change `merge.py`: the second to merge must
rebase. Revert the commit to undo.
harenome added a commit that referenced this pull request Nov 16, 2026
…l everywhere (#148)

Closes #147

## What

Each model provider is now one record that every feature reads. A
feature asks for the protocol that it speaks, and gets the providers
that serve it.

- **The registry** (`app/nullius/providers.py`). `PROTOCOLS` defines
each wire format once: its default path and how the key is sent (`chat`,
`responses`, `messages`, `systemone`, `decisions`, `openai-decisions`).
`PRESETS` holds OpenRouter, OpenAI, Anthropic and TypeSafe. `record()`
returns `{name, base, key, endpoints}` for a preset or a custom
provider. `key` is the name of the secret setting, never the key.
- **Storage** (`app/nullius/settings.py`). A record is one JSON setting
of a new kind, `provider`. A custom provider is `MODEL_<ID>`. The
endpoints added to a preset are `OPENROUTER`, `OPENAI`, `ANTHROPIC` or
`TYPESAFE`. For these settings the environment does not replace the
file. Its value is a partial object, merged over the file's value (RFC
7396, `null` removes a field). Settings shows read-only only the fields
that the environment sets. Keys keep their names and stay separate
secrets.
- **Migration.** `MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` are still
read as `name` and `base`. Saving a card moves them into the record.
`nullius migrate --providers` previews that rewrite of `settings.json`,
and `--apply` writes it. Recipes do not change. `local` and `local:<ID>`
still read as `custom`.
- **Consumers.** The registry is read by the builder, the provider test,
onboarding, `execution`, standard runs, the relay, the CLI's
`--provider` and both judges. The builder's JavaScript copy of the list
is gone. The builder offers a provider only where it serves what the
recipe's agents and judge speak, so Anthropic appears for an agent of
the standard that speaks `messages`. Review, Run (`web/execute.py`) and
`run.Run` refuse a provider that does not fit. The relay sends
`x-api-key` and `anthropic-version` on a `messages` route only, and a
bearer header on a benchmark's own routes.
- **Settings cards.** A card shows Base and a collapsed Endpoints
section with one row for each protocol. The placeholder is the default
URL. A base on a preset's host suggests that preset's endpoints. A
preset's known endpoints are fixed. An endpoint added to a preset is a
path, or a full URL on the preset's own host, because the preset's key
goes there.
- **System One** (`app/nullius/judge/client.py`) asks each decision
protocol: the TypeSafe shape on `systemone` and `decisions`, and its own
adapter for `openai-decisions`. It keeps the provider's `usage.cost` and
records a `refusal` as no answer. A 402 says what it means. A judge
never follows a redirect with the key.
- **Judge cards.** Each judge has Provider, then Protocol when the
provider serves two protocols of that judge (else one quiet line, "via
..."), then Model. Each judge's provider is a select whatever the count,
because the list is the user's own providers and grows or shrinks with
them; cards are kept for fixed sets of choices. New settings:
`LLM_JUDGE_PROTOCOL`, `SYSTEM_ONE_PROTOCOL`. On OpenRouter, System One
offers OpenRouter's public list of decision models, asked without a key,
cached 6 hours, with the fixed list as the fallback.

The seven commits are atomic and in reading order: registry and storage,
consumers, Settings cards, decision protocols, judge cards, tests, docs.

## Touches

backend (`providers`, `settings`, `migrate`, `onboarding`, `execution`,
`judge/client`, `recipes/launch`) · the standard (`run`, `relay`,
`recipe`, `cli`, `definitions`, `fidelity`) · routes (`web/settings`,
`web/execute`, `web/filters`) · templates and static (Settings cards,
judge cards, builder core) · docs (`docs/settings/model-providers.md`,
`system-one.md`, `llm-judge.md`, `README.md`, `agent-keys.md`,
`docs/builder.md`, `docs/judgement.md`, `docs/standard.md`,
`docs/cli/README.md`, `AGENTS.md`, `app/.env.example`)

## Before and after

Settings at 1280 px, light theme. Fixture keys only.

Model providers. TypeSafe is a preset like the others, so it comes after
the run providers.

| Before | After |
| --- | --- |
| <img width="748" height="431" alt="1-before-providers"
src="https://github.com/user-attachments/assets/1dd2338e-0b32-4631-8895-a6df2ba7fc29"
/> | <img width="748" height="431" alt="2-after-providers"
src="https://github.com/user-attachments/assets/ff3bb210-22ef-4bff-bbe4-df4ac4488519"
/> |

A preset card (OpenRouter), Endpoints open. Known endpoints are fixed.
Other protocols are editable rows.

| Before | After |
| --- | --- |
| <img width="748" height="360" alt="3-before-preset-card"
src="https://github.com/user-attachments/assets/5fb776e9-1c86-4cf0-8508-24041b6f9042"
/> | <img width="748" height="706" alt="4-after-preset-card"
src="https://github.com/user-attachments/assets/9058e8e4-339e-49da-8a44-67b2ffcdf8bf"
/> |

A custom provider, Endpoints open, with one endpoint set.

| Before | After |
| --- | --- |
| <img width="748" height="451" alt="5-before-custom-card"
src="https://github.com/user-attachments/assets/c725f642-3d22-4b8d-8fb0-c702e415594d"
/> | <img width="748" height="897" alt="6-after-custom-card"
src="https://github.com/user-attachments/assets/afcbcb01-fed9-4ef3-87c2-883d73594edc"
/> |

Judges. System One on OpenRouter serves two of its protocols, so it
shows Protocol. The LLM judge on OpenAI serves one, so it shows "via
OpenAI Chat Completions".

| Before | After |
| --- | --- |
| <img width="748" height="504" alt="7-before-judges"
src="https://github.com/user-attachments/assets/a089a1bd-9f72-4ca6-bff1-e1a652a83ec1"
/> | <img width="748" height="503" alt="8-after-judges"
src="https://github.com/user-attachments/assets/61ef9ae3-e080-4e54-99a1-dc43d4960ba6"
/> |

The custom card at 390 px:

<img width="358" height="1153" alt="9-after-custom-card-390"
src="https://github.com/user-attachments/assets/dce35d02-44b1-4ccb-8362-c84129e1eb87"
/>

## Verified

The branch is based on `main` at 1cc37ac. I tested its tip, each suite
in a new rootless Podman container (`python:3.12`, `--init`, uv on PATH,
Node for the route suite):

- Backend suite (`python3 app/tests/parallel.py`): 2029 tests in 118
modules, OK. `main` at 1cc37ac, in the same container: 1992 tests in
116 modules, OK.
- Route suite (`python3 app/tests/parallel.py app/ui_tests`): 441 tests
in 42 modules, OK. `main` at 1cc37ac: 431 tests in 41 modules, OK.
- New modules: `test_provider_registry` (17 tests: the merge, `null`,
partial environment values, the old settings, record checks, the preset
host rule, keys never in a record), `test_judge_decisions` (18 tests:
each decision protocol's request and reply, `usage.cost`, `refusal`,
402, no redirect, the cached model list and its fallback),
`ui_tests.test_judge_cards` (2 tests: the protocol row and the live
list). Existing modules gained tests for the relay's headers, the
builder's provider filter, Harbor recipes on Anthropic and the Settings
cards.
- End to end, against the server (`app/app.py`) and the CLI, in a
container with no network, with a new `NULLIUS_HOME` for each case and
fixture keys only. 37 checks, all pass:
  - No settings: the four presets and the OpenAI-compatible card.
- A custom provider in `settings.json`: its card, its merged record, the
default path for a protocol without an entry, System One offers it for
its `decisions` endpoint, no key in the page.
- `MODEL_GW='{"base": "https://gw-eu.example/v1", "endpoints":
{"decisions": null}}'` in the environment: the base changes, the name
stays the file's, the endpoint goes, System One no longer offers it. The
card shows Base read-only and Name editable. The file does not change.
- `nullius migrate --providers`: the preview names the moves, prints no
value and leaves the file as it was. `--apply` writes the record,
removes the old lines, keeps the key and the file mode 0600. A second
run finds nothing to move.
- Old variables in the environment (`MODEL_API_URL`, `MODEL_<ID>_NAME`,
`MODEL_<ID>_API_URL`, `LLM_JUDGE_PROVIDER=local:<ID>`): read as before.
- System One's model list on OpenRouter, offline: the route answers with
the built-in list in less than the 5 s timeout, or with a cached list,
however old.
- Browser (Chromium, Playwright), on this branch and on `main`, at 1280
px and 390 px, light and dark, with no network: the provider list, a
preset card and a custom card with Endpoints open, and both judge cards.
No page errors and no horizontal scroll.

Not verified: live calls to OpenRouter Decisions and OpenAI Decisions,
and a whole run on Anthropic. The tests use fixtures.

## Risk

A wrong record or header can send a call to the wrong URL, or with the
wrong key header. Undo with a revert. The old settings are still read,
so a revert keeps working with them. Records written by a card stay in
`settings.json` and the older code ignores them.

- **Migration on edit.** Saving a custom provider's card moves
`MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` into `MODEL_<ID>`. After a
revert, that provider loses its name and base until they are set again.
`nullius migrate --providers` without `--apply` shows what would move.
- **New reserved names.** `OPENROUTER`, `OPENAI`, `ANTHROPIC`,
`TYPESAFE` and `MODEL_<ID>` are now settings. An agent key listed under
one of these names is left out, and Settings says so. A shell variable
such as `MODEL_PATH` that is not a JSON object is not read as a record.
- **`--provider anthropic` on the CLI.** The key header now follows the
agent's protocol. An agent that speaks only `chat` is refused on
Anthropic, where it used to get `x-api-key` on OpenAI-style paths.
- **Runs and custom endpoint paths.** The relay forwards the agent's own
path below the provider's base. An Endpoints row changes where the
judges and the test ask, not where a run's calls go.
- **`openai-decisions` field names** come from OpenAI's documentation,
not from a live reply.
- **Open PRs on the same files.** This change does not depend on any of
them. Whichever merges second rebases. A test merge of this branch with
each one:
- #124 (OpenRouter serving providers, `--served-by`): conflicts in
`providers.py`, `standard/run.py`, `standard/relay.py` and
`docs/settings/model-providers.md`. Each conflict is small. Both add
functions at the same place in `providers.py`. This branch removes
`run.PROVIDERS`, where #124 adds `routing` to the OpenRouter entry: that
flag becomes a field of the OpenRouter preset. Both change
`Relay._headers`.
- #133 (stacked on #124): the same four files, and also
`standard/fidelity.py` (one `key_header=` line, now `r.key_header`) and
`docs/standard.md` (one table row each).
- #134 (stacked on #133): the same as #133, and also
`ui_tests/test_settings_page.py` (one placeholder assertion, which both
change).
- #124, #133 and #134 also conflict with `main` alone in
`standard/merge.py` (and #133, #134 in `tests/test_standard_shard.py`),
which this branch does not touch.
…on top of --served-by: gated tasks, a definition's pins as its serving pins, a pinned route's refusal as the unit's, limits of their own, and prove of a merged control

A definition's pin (setups.original.pin, score.judge.pin) joins the run's --served-by pins and wins over
--served-by for its model; ApiProviderRouteError reads the pin and reply.served_by. A failed Responses
object reads reply.end and reply.error.
…eep the score of a unit a guard stopped despite a last 429

limits.run_again false: a unit that ended rate-limited is not run once more; its outcome stands. A unit whose driver scored it after a guard's stop keeps that score, marked stopped, though its last own call got a 429.
… own last 429, not a side call's or a context window's stop

The rule reads the agent's own last call, a guard's line in the stop file and a score stopped by other than the context window; a cap's refusal is still the cap's. limits.run_again takes null as unset. The rate limits row names the opt-out.
@sirotaoleg
sirotaoleg force-pushed the feat/benchmark-pins-limits branch from d19a802 to a6d32e7 Compare November 17, 2026 13:45
@sirotaoleg
sirotaoleg marked this pull request as ready for review November 17, 2026 13:47
raise Refused(f"--served-by names {', '.join(stray)}, which this run does not call: it calls "
+ (", ".join(called) or "no model"))
self.route_pins = pins_of(bench, other, provider, self.judge_model)
self.serving.update(self.route_pins)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The definition's pins override Served by here, but the builder never shows them.

self.serving.update(self.route_pins) makes setups.original.pin and score.judge.pin win over the user's --served-by for the same model. That is the intent. However, the builder's Served by (#124: command-builder.js, recipe.py served_by) does not know these pins exist. It still says "any provider" and offers every provider for a pinned model, so the page offers a choice the run then ignores.

Reproduced with OSWorld2 (#134, which pins MiniMax-M3 to minimax and the official judge to anthropic): choose Like the paper, OpenRouter, MiniMax-M3. Served by lists 8 providers (CoreWeave, DeepInfra, GMICloud, Minimax, Parasail, ...).

Served by in the builder What the run sends
empty ("any provider") minimax only
DeepInfra minimax only, with no notice
the judge's Served by, anything (official judge) anthropic only

Review does not say the user's pin will be replaced.

Suggested fix:

  • the recipe compile (which already returns served_models) also returns each role's definition pin, computed by the same rule as pins_of;
  • the builder draws a pinned role's Served by as a fixed value (the paper's providers), with no menu;
  • the recipe check, and --served-by on the CLI, refuse a pin that disagrees with the definition's for that model, instead of overriding it silently.

This could land here or in #134. #134 is where it first becomes visible, since OSWorld2 is the first entry with a pin.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also clashes with the design of #124, which added --served-by. There, the serving provider is the user's choice: nothing is pinned unless the user asks, and an empty Served by means any provider. This PR has the definition pin some models whatever the user picks. Those are two different rules for who decides the provider.

Before choosing a fix, the authors of #124 (pierreleca) and this PR (sirotaoleg) should agree on one rule. For example:

  • the definition's pin is only a default, shown in Served by, that the user can change; or
  • the definition's pin is fixed for the Original and the official judge, and Served by shows it as fixed for those models.

aitianshi added a commit that referenced this pull request Nov 17, 2026
… work on a unit and cap its model calls (#191)

Closes #190

## What

A standard run can now force its limits, and the agent's time limit
covers its whole work on a unit.

- **Time limits by phase.** `run --timeout PHASE=SECONDS` forces the
limit of a phase: `setup`, `image`, `agent`, `verify` or `score`. Empty
keeps the default, which comes from the task, then the benchmark, then
Nullius. `config.json` records what was forced under
`standard.timeouts`.
- **One agent limit per unit.** The agent's limit is one deadline for
its whole work on a unit: every turn of a conversation, and the
simulated user's time between them. Before, it applied to each reply. A
capability keeps one limit per add and per recall, because a memory adds
a unit's sessions one by one. τ³-bench's `limits.turn_seconds` goes from
600 s a reply to 7200 s a conversation.
- **Turns.** `run --max-turns N` sets the turns of a conversation. The
builder offers it only on a benchmark that holds conversations.
- **A cap on model calls.** `run --max-calls N` caps the model calls an
agent makes on a unit. The relay counts each answered call on the
agent's route; rate-limited and failed calls are not counted, and
neither are the judge's or the simulator's. Past the cap the relay
refuses the call with a 402, and the unit ends as `UnitCallCapError`, an
agent error that is not scored. It is for agents only: a capability's
route also carries the benchmark's reader, so a capability run refuses
it.
- **A benchmark's own bounds.** They are now options marked `limit`.
τ³-bench gets Steps per conversation (`--max-steps`, from 1 up to its
default of 200) and Tool errors per conversation (`--max-errors`).
OSWorld gets Steps per task (each agent script's `--max_steps`); left
empty it uses the chosen agent's own value, declared with the new `by`
key.
- **Builder, Limits step.** It shows Scope, then Time limits in minutes,
then a section named after the benchmark when it has bounds of its own.
An empty field shows the value the run uses: in the text colour when the
benchmark or the agent declares it, grey when it is Nullius's default
(priority: Nullius < agent < benchmark). Nothing is typed into a field
for the user.

## Touches

- `app/nullius/standard/`: `run.py`, `relay.py`, `container.py`,
`job.py`, `cli.py`, `recipe.py`, `definitions.py`
- `app/nullius/runs/classify.py`: `UnitCallCapError` is an agent error
- Catalogue: τ³-bench and OSWorld's `standard.json`, their drivers and
`upstream-settings.json`
- `app/static/builder/`: the Limits step
- Docs: `docs/cli/run.md`, `docs/standard.md`, `docs/builder.md`, the
field guide and the generated supported-benchmark pages

## Before and after

The Limits step for τ³-bench, before and after:

| Before | After |
| --- | --- |
| <img width="440" alt="Limits step on main: Scope only, with tasks,
concurrency, attempts and two spending caps"
src="https://github.com/user-attachments/assets/80b468a4-326f-4057-a4b2-5551ea93e642"
/> | <img width="440" alt="Limits step on this branch: Scope with turns
and model calls, Time limits in minutes, then a tau3-bench section with
steps and tool errors"
src="https://github.com/user-attachments/assets/6278494c-6c9f-456f-b739-9578494315ee"
/> |

## Verified

- Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging
`main`, with #124's serving providers and #150's Boat VMs).
- New tests:
- A whole conversation that ends past the agent limit, though no single
turn does.
  - Four one-call turns cut on the fourth call by `max_calls=3`.
  - The relay counts only answered calls on the agent's route.
- A slow capability whose three adds each fit a 1 s limit while their
total does not.
  - A capability run refuses `--max-calls`.
- Recipe validation of the limits, including a value that rounds to 0 s.
  - τ³-bench steps above 200 are refused.
- OSWorld's `--max_steps` is replaced, or added when the agent's command
doesn't set it.
- Against the installed τ³-bench upstream: the driver uses 200 steps and
10 errors by default, and 7 and 3 when the run sets them.
- Checked in the browser, light and dark, for τ³-bench and OSWorld.

## Risk

- **Agent timeouts on conversation benchmarks.** A conversation that
used to fit 600 s per reply now has a single limit for the whole
conversation, 7200 s on τ³-bench. A very long conversation can now time
out where it didn't before. Undo by forcing a larger `--timeout
agent=SECONDS`, or by reverting `limits.turn_seconds`.
- **Comparability.** A run whose limits differ from the defaults is not
yet marked as not comparable with the leaderboard.
- **Overlap.** #174 (the agent's setup timed apart from its run) and
#133 (a benchmark's own limits) change the same parts of `run.py` and
the catalogue, so expect merge conflicts.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants