Repository navigation
feat(standard): choose the serving providers that may answer a model, in the builder and on the command line - #124
Merged
Conversation
On OpenRouter one model is served by several providers, and a run could
neither say which may answer nor see which one did. `--served-by
[MODEL=]SLUG[,SLUG...]`, repeatable, names them for a model (the run's
when MODEL= is left out). Every relay of the run, a container's sidecar
included, puts `provider: {order: [...], allow_fallbacks: false}` on each
request it sends as that model, on any route and in the three API shapes,
in place of the caller's own routing field. A pinned Chat Completions
request asks for OpenRouter's routing facts, where its reply names the
provider that served it. Without the option a request goes on byte for
byte as before.
The option is refused before any unit when the run's provider routes no
model to several (only openrouter does), when it names a model the run
does not call, or when a slug is malformed or given twice.
Each call's record keeps the routing sent (and the caller's own when it
was replaced), the serving provider the reply names, how the reply ended
and an error the reply holds under a 200, which now makes the call a
failed one. The job's config records the pin by model; each trial and
the job say the answered calls by serving provider and, for a pinned
model, whether every one was served by a pinned provider. merge refuses
shards whose pins differ, unless --force.
The run page gains the option and its rules: the slugs, what the relay sends, the refusals, the record and how a pin reads as held. The standard page names the record's new fields and the pin among merge's checks, and the providers page says that a run may choose OpenRouter's serving providers.
…ge no outcome without a pin A review found the pin's record could copy credentials, miss calls and change what a run without the option records and decides. Now: - The facts of a reply (who served it, how it ended, an error under a 200) are read by reply_facts, apart from read_reply, which is as before: the text, tool calls and usage a record held are read as they were, and no fact can raise into the delivery of a reply. Every choice is read for its end and its error, by index; a top-level error is read whatever else the reply holds; a completed Responses event's routing facts name who served it; a fact of the wrong type is left out. - An error under a 200 is recorded beside the reply and nothing else follows from it: the call keeps its status and failure reads it as before. - The recorded routing objects and facts are cleaned: a value under a credential's name, the relay's own key and key-shaped strings are removed. What is sent is unchanged. - The routing-facts header goes to OpenRouter alone, as enabled whatever the caller sent. - One check of a pin, shared by the run, its command line and the relay's own command; a control is refused a pin on a model it never calls. - Who served a unit counts its judge's and simulated user's routes, and the job the benchmark's own relay's calls; the tokens and the cost stay the agent's. A trial whose answered calls name no provider keeps its count. A name reads as a slug by its case and spaces alone. - merge compares pins as mappings, and a forced merge of different pins reads held unknown for such a model.
…held cannot tell The pages now say that an error under a 200 is recorded and changes nothing else, that the record's copies are cleaned of credentials, that who served a run counts every route, how a name reads as a slug, and that held names a provider, never its endpoint variant.
… it must refuse a pin
A run started from the builder could not pin its serving providers. On OpenRouter, the Model step now shows Served by under each model field (the model, the user simulator's and the judge's): the providers that may answer that model, tried in their order and no other, typed as slugs or added from the model's providers. Empty, the default, pins nothing. The recipe holds them as served_by, by field. The server checks them as --served-by is checked, refuses them for another provider or for a model no job calls, and compiles them into --served-by for each job that calls that model, by the name the run sends it as. Review shows them in its Model row. A saved recipe keeps them, and choosing another provider drops them. A recipe with no pin compiles to the command it did. The model's providers come from a new route, POST /new-run/endpoints, which asks OpenRouter's endpoint list as /new-run/models asks the model list: local and form-token guarded, the key kept on this machine and sent when set, the answer cached by the page for five minutes, and a fixed message when the list cannot be had, the field then taking a typed slug.
… model's providers
…del it was set for A review of the pin found records and summaries that could read held where another provider answered, credentials that could reach a record, and pins that the builder dropped or moved to another model. Now: - The provider recorded is the one of the reply's end: a stream's latest event that names one, and the last attempt of OpenRouter's routing facts; an earlier, failed attempt is never taken, and a last attempt that names none gives none. - An error keeps its code, type and message alone, of those kinds; the rest of the object is not kept. A recorded routing object's keys are cleaned as its values are. - A merged job's summary is made of each shard's whole one, its benchmark relay's calls and an unknown included. - A recipe's pins are resolved by model, then given to every job that calls that model, a control's judge job included. The compile route returns each field's model as the run sends it, and Served by lists that model's providers. - A control is refused a pin on its simulated user's model, which it never calls. relay serve refuses a pin unless its base is OpenRouter's (or --routing is given) and the model is one it sends; a container's relay is given the pins of the models it sends. - The builder clears a field's pin when its model changes, a preset or a benchmark included; keeps a saved pin as written, for the server to refuse what is wrong in sight; keeps an empty part of a typed list; puts its help in a tooltip; and keeps the keyboard on the model field once the form is drawn again.
…d where held reads unknown A variant may be a quantisation, a speed or a region. Of several choices' errors the record keeps the top-level one, else the first choice's. A provider whose name is not its slug once lowercased reads unknown though it served the call; the example of a name that holds is DeepInfra.
…facts mark as selected OpenRouter's routing facts list the endpoints available for a request and mark the one that served it as selected; they hold no list of attempts. The relay now reads that entry, at the top of a reply and in a completed Responses event's response, and asks for the facts on a pinned Responses request too: a Responses reply names its provider there alone, so a pin on a model an agent calls through Responses could never read as held.
… pin no provider serves gets 404, and why held may read unknown
…ake no name a chunk gave before
… asks for the routing facts and where held reads unknown
main gained the OSWorld1 benchmark and the run option --units since this branch started. Seven files met: the command line, the job's record, merge, the run and three pages. Both sides are kept whole, main's first: a run takes --units and --served-by together, config.json records standard.units_file and standard.served_by, and merge compares both (different unit lists are always refused, different pins unless forced).
…s record A run given --units and --served-by from the command line records both in config.json, with and without --from-run.
…n the field guide The field guide names both config fields, the routing fields of a call, the serving counters and held, and says that merge refuses different unit lists always and different pins unless forced.
main gained the settings that say what Nullius may use on each machine, with the run option --constraint, and a fix of the settings page. Seven files met: the command line, the job's record, the recipe, the run, a test module and two pages. Both sides are kept whole, main's first: a run takes --constraint and --served-by together, config.json records standard.constraints and standard.served_by, and a recipe compiles each target's constraints and the pins of the models its jobs call.
Seven cases where the two meet: both records from the command line, with and without --from-run; the order of the refusals; a scope's enforcement with a valid pin; a container relay's caps beside its pins; each target's compiled command; the builder's checks; and shards whose machine limits differ.
…ving pin in the field guide The field guide says that standard.constraints, standard.units_file and standard.served_by may stand together in config.json, and which of them a merged job keeps from its first shard.
harenome
added a commit
that referenced
this pull request
Nov 16, 2026
Closes #136 ## What On Runs → By launch, an agent or control with 2 or more shards is now one row, when `cli merge` would merge those shards. The row shows the numbers of the merged job. `launches.fold` makes that job in memory: the trials of every shard, and the job record from `merge.result_of`. `result_of` is the code that `merge` used inline to compute the job record, now a shared function. `merge.check` keeps the trial records that it already reads. - The rate is over every trial with a verdict, never an average of shard rates. - Units that no shard ran come from `merge.check` and stand beside the count ("2 of 8 not run"). They are never misses. The notes of `merge` are the tooltip. - The state is the most telling state of the shards, with a warning such as "1 of 2 shards stopped". - The header gives the dataset size, not the size of one shard. The row is folded by default. A click on the row, or Enter or Space on its chevron, unfolds one row per shard ("shard 2 · host-2"), each with its Open. The fold uses the shared chevron and slide of `disclosures.js`, and respects reduced motion. A refresh of the list keeps an unfolded row unfolded. Controls fold the same way, stay unranked and stay out of Compare. The task-check line is judged over every shard: "Task checks incomplete: oracle not run on shard 2." A participant with one shard keeps its plain row. Shards that `merge` refuses also keep their rows. Job pages, `cli merge` and merged jobs do not change. The header counts every shard ("8 runs"), every shard has its row, and each folded row says how many shards it holds ("2 shards"). A fold reads every trial record of its shards. It is kept in memory for the shard jobs that the workspace cache gives, and it is read again only when the workspace parses a shard again. A new set of shards replaces each kept fold that shares a shard with it, so the cache holds one fold per shard at most. ## Touches backend · the standard (`merge.py`) · routes · templates and static · docs and site ## Before and after One launch over 2 machines, 8 units: 2 agents, nop and oracle. Shard 2 stopped while terminus-2 ran (2 of its 4 units), and the oracle did not run on shard 2. Light theme, 1280 px. | Before | After, folded | | --- | --- | | <img width="980" height="670" alt="1-before-shards-apart" src="https://github.com/user-attachments/assets/990ceda8-8212-4003-ab1c-f380be783fc9" /> | <img width="980" height="421" alt="2-after-folded" src="https://github.com/user-attachments/assets/f37d9af1-7b3b-4b05-abc6-7dca2da1c983" /> | Before: 8 rows, 2 rates for each agent, "4 tasks", "oracle not run". After: 4 rows, "8 tasks". terminus-2 shows 50% and "3/6 · 2 of 8 not run" with "1 of 2 shards stopped". The task-check line reads "Task checks incomplete: oracle not run on shard 2." | After, unfolded (click and keyboard) | After, dark, 390 px | | --- | --- | | <img width="980" height="595" alt="3-after-unfolded" src="https://github.com/user-attachments/assets/b30f3189-d737-4658-8701-a3bbc434ad4a" /> | <img width="358" height="844" alt="4-after-dark-390" src="https://github.com/user-attachments/assets/6c4b106c-218b-4308-a934-ecac1f96934d" /> | ## Verified - Backend test `Fold.test_a_fold_reads_as_the_merged_job`: three shards, one absent, and one with a planned unit that did not run. The folded job equals `parse_job` of the `merge.merge` output on every field the row shows (solved, rates, outcomes, planned, errors, start, end, time, tokens, cost, mean score, budget). The rate is not the mean of the shard rates. The units not run and the notes are those of `merge`. Shards that `merge` refuses are not folded. - Backend test `Fold.test_a_fold_is_read_again_only_when_the_workspace_parses_a_shard_again`: a second fold of the same cached jobs does not call `merge.check`. After a trial record changes, it calls `merge.check` again. After a third shard appears, the cache holds only the fold of the three. - Route tests `test_launch_shards`: 8 shards give "8 runs", 8 shard rows and 4 folded rows that each say "2 shards". The rows are folded by default. Each shard row has its Open, and the folded row has none. The rate and time of the folded row equal those of the merged job. "2 of 8 not run" and "1 of 2 shards stopped" show. The controls fold and stay out of Compare. A nop that passes one task on shard 2 gives "A control did not behave". A stopped nop shard gives "nop stopped on shard 2". Participants with one shard keep their plain rows. - On `c7af06d5`: backend suite `python3 app/tests/parallel.py` (1989 tests) and route suite `python3 app/tests/parallel.py app/ui_tests` (328 tests) pass in a clean container. - Browser, with the fixture above: folded and unfolded, light and dark, 1280 and 390 px. No page errors and no horizontal page scroll. A click on the row unfolds it, and the shard rows slide open and closed. Tab moves the focus to the chevron of the oracle, Enter unfolds it, and Space folds it again. With reduced motion, the rows show at once. At 390 px, a long agent name wraps and the chevron stays on the line of the name. The name, its "2 shards" line and the shard rows start at the same left edge. - Render time of `/runs?view=launch`, median of 15 requests after a first one, with the same parsed jobs given to the page. 2 agents and nop and oracle, each 4 shards of 500 units (8000 trials): 71 ms before this change, 101 ms after. The first render reads the trial records once: 540 ms, against 199 ms before. The rest of the added time is the outcome bars of the 4 folded rows, over 2000 trials each. 8 shards of 1 unit each (32 trials): 11 ms before, 13 ms after. ## Risk If the fold is wrong, a row shows numbers that differ from the merged job. The backend test compares the two directly. `merge` writes the same `result.json` as before through `result_of`, and its tests pass unchanged. The first render of a large sharded launch reads all its trial records once (+0.3 s for 8000 trials). Later renders use the cache until a shard changes. The cache holds one fold per shard at most, with the parsed jobs that the workspace cache also holds. Open pull requests #124, #133 and #134 also change `merge.py`: the second to merge must rebase. Revert the commit to undo.
This was referenced Nov 16, 2026
Closed
main gained a launch's shards folded into one row on the runs page, a chosen endpoint for Hugging Face downloads, GSM8K-Platinum, a resumed run kept in its workspace, a stop on SIGTERM from any thread, and the release of v0.1.1. One file met: merge.py, where main moved the merged job's record into result_of, which the runs page's fold shares. Both sides are kept whole: result_of stays main's and its docstring says it leaves served out; merge writes the record from it and adds served, as before. A merged job's result.json is the same as before the merge.
…Served by in step with it On OpenRouter, choosing a model drew the whole Model step again and put the keyboard back on the model field after its picker had let it go: the field's menu opened again at once, holding only the chosen model, over the Served by below it. Leaving a typed model for Served by drew the step again under the pointer, so the click was lost and the keyboard went back to the model field. A typed model also cleared its pin in the recipe while Served by still showed it. A model typed or chosen now clears its field's pin and draws that Served by again where it is, its list following the new model. The rest of the step is left as it is.
…providers in order Served by sat in a block set in under each model field, a text field of slugs with a select to add from. It is now a row of its own after its model's row, in the layout of the Provider and Model rows, and its control is the Model row's picker: typing filters the model's providers, from the list the builder already fetches; a pick joins a short list under the field, in order, each removable; a slug the list lacks is taken as typed with Enter. Empty is any provider, and the help stays a tooltip. The picker takes several values in its smallest form: a field of several (list="nb-served") lists the providers of the model its field names, hands each pick to the page and is left empty for the next one, leaves out the ones already chosen, and opens again on a click. The Model field's picker is unchanged. The recipe's served_by, its checks at compile, Review's row and a pin cleared with its model are unchanged.
The builder page and the run page now say that Served by is a row after each model field's, that typing filters the model's providers, and that each pick or a slug typed and Enter joins its list, each removable.
…s while a pin changes Adding or removing a pin changed the recipe, and Served by forgot the model the server had compiled until the next compile came back. For that moment it took the name the step shows: with a paper model it listed the providers of `gpt-4o`, a name the run does not send, where the run sends `openai/gpt-4o-2024-08-06`. The model the run sends is now kept while what decides it, the recipe without its pins, is unchanged (`C.servedKey`). While it is not known, the row names no model and its picker says it is loading without asking for a list. What a row shows comes from `C.servedRow`. The rows are drawn by `servedRows`, which the tests now run under Node with stand-ins for the page's elements: a pick, a removal and a model change each draw that row again in place, and the model a row lists stays the one the run sends.
A slug with no break, 128 letters typed and taken with Enter at 390 px, ran out of its chip and out of its row. A name is never cut and it wraps. The slug now sits in a span that may shrink and breaks anywhere, and the chip and its list stay within the row's width.
…mpile fails A compile that failed was taken as the server's answer. Its error carries no models, and the row fell back to the name the step shows: after a pin, with a paper model, Served by asked for and kept the providers of `gpt-4o`, where the run sends `openai/gpt-4o-2024-08-06`. The models the server last named are now kept with what decided them, and only a compile that names models replaces them (`C.servedCompiled`). A row lists the providers of a model the server compiled and never of the name shown. Without one it waits, a typed slug still taken with Enter, and the next compile that names it ends the wait.
harenome
added a commit
that referenced
this pull request
Nov 16, 2026
…l everywhere (#148) Closes #147 ## What Each model provider is now one record that every feature reads. A feature asks for the protocol that it speaks, and gets the providers that serve it. - **The registry** (`app/nullius/providers.py`). `PROTOCOLS` defines each wire format once: its default path and how the key is sent (`chat`, `responses`, `messages`, `systemone`, `decisions`, `openai-decisions`). `PRESETS` holds OpenRouter, OpenAI, Anthropic and TypeSafe. `record()` returns `{name, base, key, endpoints}` for a preset or a custom provider. `key` is the name of the secret setting, never the key. - **Storage** (`app/nullius/settings.py`). A record is one JSON setting of a new kind, `provider`. A custom provider is `MODEL_<ID>`. The endpoints added to a preset are `OPENROUTER`, `OPENAI`, `ANTHROPIC` or `TYPESAFE`. For these settings the environment does not replace the file. Its value is a partial object, merged over the file's value (RFC 7396, `null` removes a field). Settings shows read-only only the fields that the environment sets. Keys keep their names and stay separate secrets. - **Migration.** `MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` are still read as `name` and `base`. Saving a card moves them into the record. `nullius migrate --providers` previews that rewrite of `settings.json`, and `--apply` writes it. Recipes do not change. `local` and `local:<ID>` still read as `custom`. - **Consumers.** The registry is read by the builder, the provider test, onboarding, `execution`, standard runs, the relay, the CLI's `--provider` and both judges. The builder's JavaScript copy of the list is gone. The builder offers a provider only where it serves what the recipe's agents and judge speak, so Anthropic appears for an agent of the standard that speaks `messages`. Review, Run (`web/execute.py`) and `run.Run` refuse a provider that does not fit. The relay sends `x-api-key` and `anthropic-version` on a `messages` route only, and a bearer header on a benchmark's own routes. - **Settings cards.** A card shows Base and a collapsed Endpoints section with one row for each protocol. The placeholder is the default URL. A base on a preset's host suggests that preset's endpoints. A preset's known endpoints are fixed. An endpoint added to a preset is a path, or a full URL on the preset's own host, because the preset's key goes there. - **System One** (`app/nullius/judge/client.py`) asks each decision protocol: the TypeSafe shape on `systemone` and `decisions`, and its own adapter for `openai-decisions`. It keeps the provider's `usage.cost` and records a `refusal` as no answer. A 402 says what it means. A judge never follows a redirect with the key. - **Judge cards.** Each judge has Provider, then Protocol when the provider serves two protocols of that judge (else one quiet line, "via ..."), then Model. Each judge's provider is a select whatever the count, because the list is the user's own providers and grows or shrinks with them; cards are kept for fixed sets of choices. New settings: `LLM_JUDGE_PROTOCOL`, `SYSTEM_ONE_PROTOCOL`. On OpenRouter, System One offers OpenRouter's public list of decision models, asked without a key, cached 6 hours, with the fixed list as the fallback. The seven commits are atomic and in reading order: registry and storage, consumers, Settings cards, decision protocols, judge cards, tests, docs. ## Touches backend (`providers`, `settings`, `migrate`, `onboarding`, `execution`, `judge/client`, `recipes/launch`) · the standard (`run`, `relay`, `recipe`, `cli`, `definitions`, `fidelity`) · routes (`web/settings`, `web/execute`, `web/filters`) · templates and static (Settings cards, judge cards, builder core) · docs (`docs/settings/model-providers.md`, `system-one.md`, `llm-judge.md`, `README.md`, `agent-keys.md`, `docs/builder.md`, `docs/judgement.md`, `docs/standard.md`, `docs/cli/README.md`, `AGENTS.md`, `app/.env.example`) ## Before and after Settings at 1280 px, light theme. Fixture keys only. Model providers. TypeSafe is a preset like the others, so it comes after the run providers. | Before | After | | --- | --- | | <img width="748" height="431" alt="1-before-providers" src="https://github.com/user-attachments/assets/1dd2338e-0b32-4631-8895-a6df2ba7fc29" /> | <img width="748" height="431" alt="2-after-providers" src="https://github.com/user-attachments/assets/ff3bb210-22ef-4bff-bbe4-df4ac4488519" /> | A preset card (OpenRouter), Endpoints open. Known endpoints are fixed. Other protocols are editable rows. | Before | After | | --- | --- | | <img width="748" height="360" alt="3-before-preset-card" src="https://github.com/user-attachments/assets/5fb776e9-1c86-4cf0-8508-24041b6f9042" /> | <img width="748" height="706" alt="4-after-preset-card" src="https://github.com/user-attachments/assets/9058e8e4-339e-49da-8a44-67b2ffcdf8bf" /> | A custom provider, Endpoints open, with one endpoint set. | Before | After | | --- | --- | | <img width="748" height="451" alt="5-before-custom-card" src="https://github.com/user-attachments/assets/c725f642-3d22-4b8d-8fb0-c702e415594d" /> | <img width="748" height="897" alt="6-after-custom-card" src="https://github.com/user-attachments/assets/afcbcb01-fed9-4ef3-87c2-883d73594edc" /> | Judges. System One on OpenRouter serves two of its protocols, so it shows Protocol. The LLM judge on OpenAI serves one, so it shows "via OpenAI Chat Completions". | Before | After | | --- | --- | | <img width="748" height="504" alt="7-before-judges" src="https://github.com/user-attachments/assets/a089a1bd-9f72-4ca6-bff1-e1a652a83ec1" /> | <img width="748" height="503" alt="8-after-judges" src="https://github.com/user-attachments/assets/61ef9ae3-e080-4e54-99a1-dc43d4960ba6" /> | The custom card at 390 px: <img width="358" height="1153" alt="9-after-custom-card-390" src="https://github.com/user-attachments/assets/dce35d02-44b1-4ccb-8362-c84129e1eb87" /> ## Verified The branch is based on `main` at 1cc37ac. I tested its tip, each suite in a new rootless Podman container (`python:3.12`, `--init`, uv on PATH, Node for the route suite): - Backend suite (`python3 app/tests/parallel.py`): 2029 tests in 118 modules, OK. `main` at 1cc37ac, in the same container: 1992 tests in 116 modules, OK. - Route suite (`python3 app/tests/parallel.py app/ui_tests`): 441 tests in 42 modules, OK. `main` at 1cc37ac: 431 tests in 41 modules, OK. - New modules: `test_provider_registry` (17 tests: the merge, `null`, partial environment values, the old settings, record checks, the preset host rule, keys never in a record), `test_judge_decisions` (18 tests: each decision protocol's request and reply, `usage.cost`, `refusal`, 402, no redirect, the cached model list and its fallback), `ui_tests.test_judge_cards` (2 tests: the protocol row and the live list). Existing modules gained tests for the relay's headers, the builder's provider filter, Harbor recipes on Anthropic and the Settings cards. - End to end, against the server (`app/app.py`) and the CLI, in a container with no network, with a new `NULLIUS_HOME` for each case and fixture keys only. 37 checks, all pass: - No settings: the four presets and the OpenAI-compatible card. - A custom provider in `settings.json`: its card, its merged record, the default path for a protocol without an entry, System One offers it for its `decisions` endpoint, no key in the page. - `MODEL_GW='{"base": "https://gw-eu.example/v1", "endpoints": {"decisions": null}}'` in the environment: the base changes, the name stays the file's, the endpoint goes, System One no longer offers it. The card shows Base read-only and Name editable. The file does not change. - `nullius migrate --providers`: the preview names the moves, prints no value and leaves the file as it was. `--apply` writes the record, removes the old lines, keeps the key and the file mode 0600. A second run finds nothing to move. - Old variables in the environment (`MODEL_API_URL`, `MODEL_<ID>_NAME`, `MODEL_<ID>_API_URL`, `LLM_JUDGE_PROVIDER=local:<ID>`): read as before. - System One's model list on OpenRouter, offline: the route answers with the built-in list in less than the 5 s timeout, or with a cached list, however old. - Browser (Chromium, Playwright), on this branch and on `main`, at 1280 px and 390 px, light and dark, with no network: the provider list, a preset card and a custom card with Endpoints open, and both judge cards. No page errors and no horizontal scroll. Not verified: live calls to OpenRouter Decisions and OpenAI Decisions, and a whole run on Anthropic. The tests use fixtures. ## Risk A wrong record or header can send a call to the wrong URL, or with the wrong key header. Undo with a revert. The old settings are still read, so a revert keeps working with them. Records written by a card stay in `settings.json` and the older code ignores them. - **Migration on edit.** Saving a custom provider's card moves `MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` into `MODEL_<ID>`. After a revert, that provider loses its name and base until they are set again. `nullius migrate --providers` without `--apply` shows what would move. - **New reserved names.** `OPENROUTER`, `OPENAI`, `ANTHROPIC`, `TYPESAFE` and `MODEL_<ID>` are now settings. An agent key listed under one of these names is left out, and Settings says so. A shell variable such as `MODEL_PATH` that is not a JSON object is not read as a record. - **`--provider anthropic` on the CLI.** The key header now follows the agent's protocol. An agent that speaks only `chat` is refused on Anthropic, where it used to get `x-api-key` on OpenAI-style paths. - **Runs and custom endpoint paths.** The relay forwards the agent's own path below the provider's base. An Endpoints row changes where the judges and the test ask, not where a run's calls go. - **`openai-decisions` field names** come from OpenAI's documentation, not from a live reply. - **Open PRs on the same files.** This change does not depend on any of them. Whichever merges second rebases. A test merge of this branch with each one: - #124 (OpenRouter serving providers, `--served-by`): conflicts in `providers.py`, `standard/run.py`, `standard/relay.py` and `docs/settings/model-providers.md`. Each conflict is small. Both add functions at the same place in `providers.py`. This branch removes `run.PROVIDERS`, where #124 adds `routing` to the OpenRouter entry: that flag becomes a field of the OpenRouter preset. Both change `Relay._headers`. - #133 (stacked on #124): the same four files, and also `standard/fidelity.py` (one `key_header=` line, now `r.key_header`) and `docs/standard.md` (one table row each). - #134 (stacked on #133): the same as #133, and also `ui_tests/test_settings_page.py` (one placeholder assertion, which both change). - #124, #133 and #134 also conflict with `main` alone in `standard/merge.py` (and #133, #134 in `tests/test_standard_shard.py`), which this branch does not touch.
Accept named models for the current recipe when only provider pins differ. Keep the full recipe key check for page output and hide cached models after a model change. Exercise overlapping successful and failed compile answers in both orders.
# Conflicts: # app/nullius/providers.py # app/nullius/standard/relay.py # app/nullius/standard/run.py # docs/settings/model-providers.md
# Conflicts: # app/nullius/standard/recipe.py
Ask for serving providers when compilation names the model sent by each field. Share the five-minute cache with the picker and update each row in place. Hide an unpinned row while its list is pending or contains one entry. Keep pinned rows and empty or failed lists visible. Ignore responses for models that have changed. Cover visibility, request reuse and model changes under Node. Update the builder and provider guides.
The list of serving providers was already asked through the page's request helper, passed in under the name fetch. The check that every request of a page goes through the helper reads the scripts' text and took that name for a bare call. The parameter is now named request.
aitianshi
approved these changes
Nov 17, 2026
aitianshi
added a commit
that referenced
this pull request
Nov 17, 2026
… work on a unit and cap its model calls (#191) Closes #190 ## What A standard run can now force its limits, and the agent's time limit covers its whole work on a unit. - **Time limits by phase.** `run --timeout PHASE=SECONDS` forces the limit of a phase: `setup`, `image`, `agent`, `verify` or `score`. Empty keeps the default, which comes from the task, then the benchmark, then Nullius. `config.json` records what was forced under `standard.timeouts`. - **One agent limit per unit.** The agent's limit is one deadline for its whole work on a unit: every turn of a conversation, and the simulated user's time between them. Before, it applied to each reply. A capability keeps one limit per add and per recall, because a memory adds a unit's sessions one by one. τ³-bench's `limits.turn_seconds` goes from 600 s a reply to 7200 s a conversation. - **Turns.** `run --max-turns N` sets the turns of a conversation. The builder offers it only on a benchmark that holds conversations. - **A cap on model calls.** `run --max-calls N` caps the model calls an agent makes on a unit. The relay counts each answered call on the agent's route; rate-limited and failed calls are not counted, and neither are the judge's or the simulator's. Past the cap the relay refuses the call with a 402, and the unit ends as `UnitCallCapError`, an agent error that is not scored. It is for agents only: a capability's route also carries the benchmark's reader, so a capability run refuses it. - **A benchmark's own bounds.** They are now options marked `limit`. τ³-bench gets Steps per conversation (`--max-steps`, from 1 up to its default of 200) and Tool errors per conversation (`--max-errors`). OSWorld gets Steps per task (each agent script's `--max_steps`); left empty it uses the chosen agent's own value, declared with the new `by` key. - **Builder, Limits step.** It shows Scope, then Time limits in minutes, then a section named after the benchmark when it has bounds of its own. An empty field shows the value the run uses: in the text colour when the benchmark or the agent declares it, grey when it is Nullius's default (priority: Nullius < agent < benchmark). Nothing is typed into a field for the user. ## Touches - `app/nullius/standard/`: `run.py`, `relay.py`, `container.py`, `job.py`, `cli.py`, `recipe.py`, `definitions.py` - `app/nullius/runs/classify.py`: `UnitCallCapError` is an agent error - Catalogue: τ³-bench and OSWorld's `standard.json`, their drivers and `upstream-settings.json` - `app/static/builder/`: the Limits step - Docs: `docs/cli/run.md`, `docs/standard.md`, `docs/builder.md`, the field guide and the generated supported-benchmark pages ## Before and after The Limits step for τ³-bench, before and after: | Before | After | | --- | --- | | <img width="440" alt="Limits step on main: Scope only, with tasks, concurrency, attempts and two spending caps" src="https://github.com/user-attachments/assets/80b468a4-326f-4057-a4b2-5551ea93e642" /> | <img width="440" alt="Limits step on this branch: Scope with turns and model calls, Time limits in minutes, then a tau3-bench section with steps and tool errors" src="https://github.com/user-attachments/assets/6278494c-6c9f-456f-b739-9578494315ee" /> | ## Verified - Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging `main`, with #124's serving providers and #150's Boat VMs). - New tests: - A whole conversation that ends past the agent limit, though no single turn does. - Four one-call turns cut on the fourth call by `max_calls=3`. - The relay counts only answered calls on the agent's route. - A slow capability whose three adds each fit a 1 s limit while their total does not. - A capability run refuses `--max-calls`. - Recipe validation of the limits, including a value that rounds to 0 s. - τ³-bench steps above 200 are refused. - OSWorld's `--max_steps` is replaced, or added when the agent's command doesn't set it. - Against the installed τ³-bench upstream: the driver uses 200 steps and 10 errors by default, and 7 and 3 when the run sets them. - Checked in the browser, light and dark, for τ³-bench and OSWorld. ## Risk - **Agent timeouts on conversation benchmarks.** A conversation that used to fit 600 s per reply now has a single limit for the whole conversation, 7200 s on τ³-bench. A very long conversation can now time out where it didn't before. Undo by forcing a larger `--timeout agent=SECONDS`, or by reverting `limits.turn_seconds`. - **Comparability.** A run whose limits differ from the defaults is not yet marked as not comparable with the leaderboard. - **Overlap.** #174 (the agent's setup timed apart from its run) and #133 (a benchmark's own limits) change the same parts of `run.py` and the catalogue, so expect merge conflicts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #123
What
On OpenRouter one model is served by several providers, and a run could neither choose among them nor see which one answered. This adds a Served by choice:
--served-by [MODEL=]SLUG[,SLUG...]onnullius standard runandrelay serve, and a row after each model field of the builder's Model step (not shown for a model with a single serving provider). The relay puts OpenRouter's strict pin (provider: {"order": [...], "allow_fallbacks": false}) on every request it sends as a pinned model. Each call's record now names the provider that served it, how the reply ended and an error inside a 200. Each trial and job counts its answered calls by serving provider and says whether the pin held, andmergerefuses shards whose pins differ. A pin is taken for OpenRouter alone and refused before any unit otherwise. Without the option, requests, records and outcomes are unchanged.The pages have the detail:
docs/cli/run.md,docs/builder.md,docs/standard.md,docs/settings/model-providers.md.One limit: a reply names its provider by display name. A pin on a provider whose name differs from its slug (
Googleforgoogle-vertex) readsheld: unknown.Left for later: a quantisation filter, a default in Settings, and counting an error inside a 200 as a failed call.
Touches
backend and the standard · routes · templates and static · docs and site
Before and after
The builder's Model step on OpenRouter, light theme, 1280 px. Each picture opens at full size.
Verified
main. The branch's four new backend modules (62 tests) and its builder module (28 tests) fail onmain.mainand one of this branch sent identical bytes for the same 36 requests, and four recipes with no pin compile to the same command.d6f65146(relay.pyis unchanged since), in ten small requests throughrelay serve: a pin was served by its provider in the three API shapes, streamed or not, and recorded; a pin no provider can serve was answered 404 with nothing charged.Risk
Every model call of a run passes through the relay. Without
--served-bynothing changes (the comparisons above). With a pin, OpenRouter answers 404 when no pinned provider can serve the request, and the call fails as any refused call does. A recipe saved with a pin and opened by an older Nullius loses the pin. Undo with a revert.