Skip to content

feat(standard): choose the serving providers that may answer a model, in the builder and on the command line - #124

Merged
aitianshi merged 35 commits into
ThinkFlowLab:mainfrom
pierreleca:feat/served-by
Nov 17, 2026
Merged

aitianshi merged 35 commits into
ThinkFlowLab:mainfrom
pierreleca:feat/served-by

Conversation

@pierreleca

@pierreleca pierreleca commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Closes #123

What

On OpenRouter one model is served by several providers, and a run could neither choose among them nor see which one answered. This adds a Served by choice: --served-by [MODEL=]SLUG[,SLUG...] on nullius standard run and relay serve, and a row after each model field of the builder's Model step (not shown for a model with a single serving provider). The relay puts OpenRouter's strict pin (provider: {"order": [...], "allow_fallbacks": false}) on every request it sends as a pinned model. Each call's record now names the provider that served it, how the reply ended and an error inside a 200. Each trial and job counts its answered calls by serving provider and says whether the pin held, and merge refuses shards whose pins differ. A pin is taken for OpenRouter alone and refused before any unit otherwise. Without the option, requests, records and outcomes are unchanged.

The pages have the detail: docs/cli/run.md, docs/builder.md, docs/standard.md, docs/settings/model-providers.md.

One limit: a reply names its provider by display name. A pin on a provider whose name differs from its slug (Google for google-vertex) reads held: unknown.

Left for later: a quantisation filter, a default in Settings, and counting an error inside a 200 as a failed call.

Touches

backend and the standard · routes · templates and static · docs and site

Before and after

The builder's Model step on OpenRouter, light theme, 1280 px. Each picture opens at full size.

Before After
The Model step on main The Model step with Served by rows

Verified

  • Tests, each module in its own process, offline: on the branch 61 of 62 backend modules and 17 of 17 route modules pass, and the one failing module fails the same way on main. The branch's four new backend modules (62 tests) and its builder module (28 tests) fail on main.
  • Without a pin: a relay of main and one of this branch sent identical bytes for the same 36 requests, and four recipes with no pin compile to the same command.
  • In a browser, with a stand-in for OpenRouter: the Model step's flow and Review, in the light and dark themes at 1280 px and at 390 px, with no page error.
  • Against OpenRouter itself, at d6f65146 (relay.py is unchanged since), in ten small requests through relay serve: a pin was served by its provider in the three API shapes, streamed or not, and recorded; a pin no provider can serve was answered 404 with nothing charged.
  • Not verified: a whole benchmark run against OpenRouter, the docker tier, the rest of the suites.

Risk

Every model call of a run passes through the relay. Without --served-by nothing changes (the comparisons above). With a pin, OpenRouter answers 404 when no pinned provider can serve the request, and the call fails as any refused call does. A recipe saved with a pin and opened by an older Nullius loses the pin. Undo with a revert.

On OpenRouter one model is served by several providers, and a run could
neither say which may answer nor see which one did. `--served-by
[MODEL=]SLUG[,SLUG...]`, repeatable, names them for a model (the run's
when MODEL= is left out). Every relay of the run, a container's sidecar
included, puts `provider: {order: [...], allow_fallbacks: false}` on each
request it sends as that model, on any route and in the three API shapes,
in place of the caller's own routing field. A pinned Chat Completions
request asks for OpenRouter's routing facts, where its reply names the
provider that served it. Without the option a request goes on byte for
byte as before.

The option is refused before any unit when the run's provider routes no
model to several (only openrouter does), when it names a model the run
does not call, or when a slug is malformed or given twice.

Each call's record keeps the routing sent (and the caller's own when it
was replaced), the serving provider the reply names, how the reply ended
and an error the reply holds under a 200, which now makes the call a
failed one. The job's config records the pin by model; each trial and
the job say the answered calls by serving provider and, for a pinned
model, whether every one was served by a pinned provider. merge refuses
shards whose pins differ, unless --force.
The run page gains the option and its rules: the slugs, what the relay
sends, the refusals, the record and how a pin reads as held. The standard
page names the record's new fields and the pin among merge's checks, and
the providers page says that a run may choose OpenRouter's serving
providers.
…ge no outcome without a pin

A review found the pin's record could copy credentials, miss calls and
change what a run without the option records and decides. Now:

- The facts of a reply (who served it, how it ended, an error under a
  200) are read by reply_facts, apart from read_reply, which is as
  before: the text, tool calls and usage a record held are read as they
  were, and no fact can raise into the delivery of a reply. Every choice
  is read for its end and its error, by index; a top-level error is read
  whatever else the reply holds; a completed Responses event's routing
  facts name who served it; a fact of the wrong type is left out.
- An error under a 200 is recorded beside the reply and nothing else
  follows from it: the call keeps its status and failure reads it as
  before.
- The recorded routing objects and facts are cleaned: a value under a
  credential's name, the relay's own key and key-shaped strings are
  removed. What is sent is unchanged.
- The routing-facts header goes to OpenRouter alone, as enabled whatever
  the caller sent.
- One check of a pin, shared by the run, its command line and the relay's
  own command; a control is refused a pin on a model it never calls.
- Who served a unit counts its judge's and simulated user's routes, and
  the job the benchmark's own relay's calls; the tokens and the cost stay
  the agent's. A trial whose answered calls name no provider keeps its
  count. A name reads as a slug by its case and spaces alone.
- merge compares pins as mappings, and a forced merge of different pins
  reads held unknown for such a model.
…held cannot tell

The pages now say that an error under a 200 is recorded and changes
nothing else, that the record's copies are cleaned of credentials, that
who served a run counts every route, how a name reads as a slug, and that
held names a provider, never its endpoint variant.
A run started from the builder could not pin its serving providers. On
OpenRouter, the Model step now shows Served by under each model field
(the model, the user simulator's and the judge's): the providers that
may answer that model, tried in their order and no other, typed as slugs
or added from the model's providers. Empty, the default, pins nothing.

The recipe holds them as served_by, by field. The server checks them as
--served-by is checked, refuses them for another provider or for a model
no job calls, and compiles them into --served-by for each job that calls
that model, by the name the run sends it as. Review shows them in its
Model row. A saved recipe keeps them, and choosing another provider drops
them. A recipe with no pin compiles to the command it did.

The model's providers come from a new route, POST /new-run/endpoints,
which asks OpenRouter's endpoint list as /new-run/models asks the model
list: local and form-token guarded, the key kept on this machine and sent
when set, the answer cached by the page for five minutes, and a fixed
message when the list cannot be had, the field then taking a typed slug.
…del it was set for

A review of the pin found records and summaries that could read held
where another provider answered, credentials that could reach a record,
and pins that the builder dropped or moved to another model. Now:

- The provider recorded is the one of the reply's end: a stream's latest
  event that names one, and the last attempt of OpenRouter's routing
  facts; an earlier, failed attempt is never taken, and a last attempt
  that names none gives none.
- An error keeps its code, type and message alone, of those kinds; the
  rest of the object is not kept. A recorded routing object's keys are
  cleaned as its values are.
- A merged job's summary is made of each shard's whole one, its
  benchmark relay's calls and an unknown included.
- A recipe's pins are resolved by model, then given to every job that
  calls that model, a control's judge job included. The compile route
  returns each field's model as the run sends it, and Served by lists
  that model's providers.
- A control is refused a pin on its simulated user's model, which it
  never calls. relay serve refuses a pin unless its base is OpenRouter's
  (or --routing is given) and the model is one it sends; a container's
  relay is given the pins of the models it sends.
- The builder clears a field's pin when its model changes, a preset or a
  benchmark included; keeps a saved pin as written, for the server to
  refuse what is wrong in sight; keeps an empty part of a typed list; puts
  its help in a tooltip; and keeps the keyboard on the model field once
  the form is drawn again.
…d where held reads unknown

A variant may be a quantisation, a speed or a region. Of several
choices' errors the record keeps the top-level one, else the first
choice's. A provider whose name is not its slug once lowercased reads
unknown though it served the call; the example of a name that holds is
DeepInfra.
…facts mark as selected

OpenRouter's routing facts list the endpoints available for a request and
mark the one that served it as selected; they hold no list of attempts.
The relay now reads that entry, at the top of a reply and in a completed
Responses event's response, and asks for the facts on a pinned Responses
request too: a Responses reply names its provider there alone, so a pin
on a model an agent calls through Responses could never read as held.
… pin no provider serves gets 404, and why held may read unknown
… asks for the routing facts and where held reads unknown
main gained the OSWorld1 benchmark and the run option --units since this branch started. Seven files met: the command line, the job's record, merge, the run and three pages. Both sides are kept whole, main's first: a run takes --units and --served-by together, config.json records standard.units_file and standard.served_by, and merge compares both (different unit lists are always refused, different pins unless forced).
…s record

A run given --units and --served-by from the command line records both in config.json, with and without --from-run.
…n the field guide

The field guide names both config fields, the routing fields of a call, the serving counters and held, and says that merge refuses different unit lists always and different pins unless forced.
main gained the settings that say what Nullius may use on each machine, with the run option --constraint, and a fix of the settings page. Seven files met: the command line, the job's record, the recipe, the run, a test module and two pages. Both sides are kept whole, main's first: a run takes --constraint and --served-by together, config.json records standard.constraints and standard.served_by, and a recipe compiles each target's constraints and the pins of the models its jobs call.
Seven cases where the two meet: both records from the command line, with and without --from-run; the order of the refusals; a scope's enforcement with a valid pin; a container relay's caps beside its pins; each target's compiled command; the builder's checks; and shards whose machine limits differ.
…ving pin in the field guide

The field guide says that standard.constraints, standard.units_file and standard.served_by may stand together in config.json, and which of them a merged job keeps from its first shard.
harenome added a commit that referenced this pull request Nov 16, 2026
Closes #136

## What

On Runs → By launch, an agent or control with 2 or more shards is now
one row, when `cli merge` would merge those shards. The row shows the
numbers of the merged job. `launches.fold` makes that job in memory: the
trials of every shard, and the job record from `merge.result_of`.
`result_of` is the code that `merge` used inline to compute the job
record, now a shared function. `merge.check` keeps the trial records
that it already reads.

- The rate is over every trial with a verdict, never an average of shard
rates.
- Units that no shard ran come from `merge.check` and stand beside the
count ("2 of 8 not run"). They are never misses. The notes of `merge`
are the tooltip.
- The state is the most telling state of the shards, with a warning such
as "1 of 2 shards stopped".
- The header gives the dataset size, not the size of one shard.

The row is folded by default. A click on the row, or Enter or Space on
its chevron, unfolds one row per shard ("shard 2 · host-2"), each with
its Open. The fold uses the shared chevron and slide of
`disclosures.js`, and respects reduced motion. A refresh of the list
keeps an unfolded row unfolded. Controls fold the same way, stay
unranked and stay out of Compare. The task-check line is judged over
every shard: "Task checks incomplete: oracle not run on shard 2."

A participant with one shard keeps its plain row. Shards that `merge`
refuses also keep their rows. Job pages, `cli merge` and merged jobs do
not change. The header counts every shard ("8 runs"), every shard has
its row, and each folded row says how many shards it holds ("2 shards").

A fold reads every trial record of its shards. It is kept in memory for
the shard jobs that the workspace cache gives, and it is read again only
when the workspace parses a shard again. A new set of shards replaces
each kept fold that shares a shard with it, so the cache holds one fold
per shard at most.

## Touches

backend · the standard (`merge.py`) · routes · templates and static ·
docs and site

## Before and after

One launch over 2 machines, 8 units: 2 agents, nop and oracle. Shard 2
stopped while terminus-2 ran (2 of its 4 units), and the oracle did not
run on shard 2. Light theme, 1280 px.

| Before | After, folded |
| --- | --- |
| <img width="980" height="670" alt="1-before-shards-apart"
src="https://github.com/user-attachments/assets/990ceda8-8212-4003-ab1c-f380be783fc9"
/> | <img width="980" height="421" alt="2-after-folded"
src="https://github.com/user-attachments/assets/f37d9af1-7b3b-4b05-abc6-7dca2da1c983"
/> |

Before: 8 rows, 2 rates for each agent, "4 tasks", "oracle not run".
After: 4 rows, "8 tasks". terminus-2 shows 50% and "3/6 · 2 of 8 not
run" with "1 of 2 shards stopped". The task-check line reads "Task
checks incomplete: oracle not run on shard 2."

| After, unfolded (click and keyboard) | After, dark, 390 px |
| --- | --- |
| <img width="980" height="595" alt="3-after-unfolded"
src="https://github.com/user-attachments/assets/b30f3189-d737-4658-8701-a3bbc434ad4a"
/> | <img width="358" height="844" alt="4-after-dark-390"
src="https://github.com/user-attachments/assets/6c4b106c-218b-4308-a934-ecac1f96934d"
/> |

## Verified

- Backend test `Fold.test_a_fold_reads_as_the_merged_job`: three shards,
one absent, and one with a planned unit that did not run. The folded job
equals `parse_job` of the `merge.merge` output on every field the row
shows (solved, rates, outcomes, planned, errors, start, end, time,
tokens, cost, mean score, budget). The rate is not the mean of the shard
rates. The units not run and the notes are those of `merge`. Shards that
`merge` refuses are not folded.
- Backend test
`Fold.test_a_fold_is_read_again_only_when_the_workspace_parses_a_shard_again`:
a second fold of the same cached jobs does not call `merge.check`. After
a trial record changes, it calls `merge.check` again. After a third
shard appears, the cache holds only the fold of the three.
- Route tests `test_launch_shards`: 8 shards give "8 runs", 8 shard rows
and 4 folded rows that each say "2 shards". The rows are folded by
default. Each shard row has its Open, and the folded row has none. The
rate and time of the folded row equal those of the merged job. "2 of 8
not run" and "1 of 2 shards stopped" show. The controls fold and stay
out of Compare. A nop that passes one task on shard 2 gives "A control
did not behave". A stopped nop shard gives "nop stopped on shard 2".
Participants with one shard keep their plain rows.
- On `c7af06d5`: backend suite `python3 app/tests/parallel.py` (1989
tests) and route suite `python3 app/tests/parallel.py app/ui_tests` (328
tests) pass in a clean container.
- Browser, with the fixture above: folded and unfolded, light and dark,
1280 and 390 px. No page errors and no horizontal page scroll. A click
on the row unfolds it, and the shard rows slide open and closed. Tab
moves the focus to the chevron of the oracle, Enter unfolds it, and
Space folds it again. With reduced motion, the rows show at once. At 390
px, a long agent name wraps and the chevron stays on the line of the
name. The name, its "2 shards" line and the shard rows start at the same
left edge.
- Render time of `/runs?view=launch`, median of 15 requests after a
first one, with the same parsed jobs given to the page. 2 agents and nop
and oracle, each 4 shards of 500 units (8000 trials): 71 ms before this
change, 101 ms after. The first render reads the trial records once: 540
ms, against 199 ms before. The rest of the added time is the outcome
bars of the 4 folded rows, over 2000 trials each. 8 shards of 1 unit
each (32 trials): 11 ms before, 13 ms after.

## Risk

If the fold is wrong, a row shows numbers that differ from the merged
job. The backend test compares the two directly. `merge` writes the same
`result.json` as before through `result_of`, and its tests pass
unchanged. The first render of a large sharded launch reads all its
trial records once (+0.3 s for 8000 trials). Later renders use the cache
until a shard changes. The cache holds one fold per shard at most, with
the parsed jobs that the workspace cache also holds. Open pull requests
#124, #133 and #134 also change `merge.py`: the second to merge must
rebase. Revert the commit to undo.
main gained a launch's shards folded into one row on the runs page, a chosen endpoint for Hugging Face downloads, GSM8K-Platinum, a resumed run kept in its workspace, a stop on SIGTERM from any thread, and the release of v0.1.1. One file met: merge.py, where main moved the merged job's record into result_of, which the runs page's fold shares. Both sides are kept whole: result_of stays main's and its docstring says it leaves served out; merge writes the record from it and adds served, as before. A merged job's result.json is the same as before the merge.
…Served by in step with it

On OpenRouter, choosing a model drew the whole Model step again and put the keyboard back on the model field after its picker had let it go: the field's menu opened again at once, holding only the chosen model, over the Served by below it. Leaving a typed model for Served by drew the step again under the pointer, so the click was lost and the keyboard went back to the model field. A typed model also cleared its pin in the recipe while Served by still showed it.

A model typed or chosen now clears its field's pin and draws that Served by again where it is, its list following the new model. The rest of the step is left as it is.
…providers in order

Served by sat in a block set in under each model field, a text field of slugs with a select to add from. It is now a row of its own after its model's row, in the layout of the Provider and Model rows, and its control is the Model row's picker: typing filters the model's providers, from the list the builder already fetches; a pick joins a short list under the field, in order, each removable; a slug the list lacks is taken as typed with Enter. Empty is any provider, and the help stays a tooltip.

The picker takes several values in its smallest form: a field of several (list="nb-served") lists the providers of the model its field names, hands each pick to the page and is left empty for the next one, leaves out the ones already chosen, and opens again on a click. The Model field's picker is unchanged. The recipe's served_by, its checks at compile, Review's row and a pin cleared with its model are unchanged.
The builder page and the run page now say that Served by is a row after each model field's, that typing filters the model's providers, and that each pick or a slug typed and Enter joins its list, each removable.
…s while a pin changes

Adding or removing a pin changed the recipe, and Served by forgot the model the server had compiled until the next compile came back. For that moment it took the name the step shows: with a paper model it listed the providers of `gpt-4o`, a name the run does not send, where the run sends `openai/gpt-4o-2024-08-06`.

The model the run sends is now kept while what decides it, the recipe without its pins, is unchanged (`C.servedKey`). While it is not known, the row names no model and its picker says it is loading without asking for a list. What a row shows comes from `C.servedRow`. The rows are drawn by `servedRows`, which the tests now run under Node with stand-ins for the page's elements: a pick, a removal and a model change each draw that row again in place, and the model a row lists stays the one the run sends.
A slug with no break, 128 letters typed and taken with Enter at 390 px, ran out of its chip and out of its row. A name is never cut and it wraps. The slug now sits in a span that may shrink and breaks anywhere, and the chip and its list stay within the row's width.
…mpile fails

A compile that failed was taken as the server's answer. Its error carries no models, and the row fell back to the name the step shows: after a pin, with a paper model, Served by asked for and kept the providers of `gpt-4o`, where the run sends `openai/gpt-4o-2024-08-06`.

The models the server last named are now kept with what decided them, and only a compile that names models replaces them (`C.servedCompiled`). A row lists the providers of a model the server compiled and never of the name shown. Without one it waits, a typed slug still taken with Enter, and the next compile that names it ends the wait.
harenome added a commit that referenced this pull request Nov 16, 2026
…l everywhere (#148)

Closes #147

## What

Each model provider is now one record that every feature reads. A
feature asks for the protocol that it speaks, and gets the providers
that serve it.

- **The registry** (`app/nullius/providers.py`). `PROTOCOLS` defines
each wire format once: its default path and how the key is sent (`chat`,
`responses`, `messages`, `systemone`, `decisions`, `openai-decisions`).
`PRESETS` holds OpenRouter, OpenAI, Anthropic and TypeSafe. `record()`
returns `{name, base, key, endpoints}` for a preset or a custom
provider. `key` is the name of the secret setting, never the key.
- **Storage** (`app/nullius/settings.py`). A record is one JSON setting
of a new kind, `provider`. A custom provider is `MODEL_<ID>`. The
endpoints added to a preset are `OPENROUTER`, `OPENAI`, `ANTHROPIC` or
`TYPESAFE`. For these settings the environment does not replace the
file. Its value is a partial object, merged over the file's value (RFC
7396, `null` removes a field). Settings shows read-only only the fields
that the environment sets. Keys keep their names and stay separate
secrets.
- **Migration.** `MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` are still
read as `name` and `base`. Saving a card moves them into the record.
`nullius migrate --providers` previews that rewrite of `settings.json`,
and `--apply` writes it. Recipes do not change. `local` and `local:<ID>`
still read as `custom`.
- **Consumers.** The registry is read by the builder, the provider test,
onboarding, `execution`, standard runs, the relay, the CLI's
`--provider` and both judges. The builder's JavaScript copy of the list
is gone. The builder offers a provider only where it serves what the
recipe's agents and judge speak, so Anthropic appears for an agent of
the standard that speaks `messages`. Review, Run (`web/execute.py`) and
`run.Run` refuse a provider that does not fit. The relay sends
`x-api-key` and `anthropic-version` on a `messages` route only, and a
bearer header on a benchmark's own routes.
- **Settings cards.** A card shows Base and a collapsed Endpoints
section with one row for each protocol. The placeholder is the default
URL. A base on a preset's host suggests that preset's endpoints. A
preset's known endpoints are fixed. An endpoint added to a preset is a
path, or a full URL on the preset's own host, because the preset's key
goes there.
- **System One** (`app/nullius/judge/client.py`) asks each decision
protocol: the TypeSafe shape on `systemone` and `decisions`, and its own
adapter for `openai-decisions`. It keeps the provider's `usage.cost` and
records a `refusal` as no answer. A 402 says what it means. A judge
never follows a redirect with the key.
- **Judge cards.** Each judge has Provider, then Protocol when the
provider serves two protocols of that judge (else one quiet line, "via
..."), then Model. Each judge's provider is a select whatever the count,
because the list is the user's own providers and grows or shrinks with
them; cards are kept for fixed sets of choices. New settings:
`LLM_JUDGE_PROTOCOL`, `SYSTEM_ONE_PROTOCOL`. On OpenRouter, System One
offers OpenRouter's public list of decision models, asked without a key,
cached 6 hours, with the fixed list as the fallback.

The seven commits are atomic and in reading order: registry and storage,
consumers, Settings cards, decision protocols, judge cards, tests, docs.

## Touches

backend (`providers`, `settings`, `migrate`, `onboarding`, `execution`,
`judge/client`, `recipes/launch`) · the standard (`run`, `relay`,
`recipe`, `cli`, `definitions`, `fidelity`) · routes (`web/settings`,
`web/execute`, `web/filters`) · templates and static (Settings cards,
judge cards, builder core) · docs (`docs/settings/model-providers.md`,
`system-one.md`, `llm-judge.md`, `README.md`, `agent-keys.md`,
`docs/builder.md`, `docs/judgement.md`, `docs/standard.md`,
`docs/cli/README.md`, `AGENTS.md`, `app/.env.example`)

## Before and after

Settings at 1280 px, light theme. Fixture keys only.

Model providers. TypeSafe is a preset like the others, so it comes after
the run providers.

| Before | After |
| --- | --- |
| <img width="748" height="431" alt="1-before-providers"
src="https://github.com/user-attachments/assets/1dd2338e-0b32-4631-8895-a6df2ba7fc29"
/> | <img width="748" height="431" alt="2-after-providers"
src="https://github.com/user-attachments/assets/ff3bb210-22ef-4bff-bbe4-df4ac4488519"
/> |

A preset card (OpenRouter), Endpoints open. Known endpoints are fixed.
Other protocols are editable rows.

| Before | After |
| --- | --- |
| <img width="748" height="360" alt="3-before-preset-card"
src="https://github.com/user-attachments/assets/5fb776e9-1c86-4cf0-8508-24041b6f9042"
/> | <img width="748" height="706" alt="4-after-preset-card"
src="https://github.com/user-attachments/assets/9058e8e4-339e-49da-8a44-67b2ffcdf8bf"
/> |

A custom provider, Endpoints open, with one endpoint set.

| Before | After |
| --- | --- |
| <img width="748" height="451" alt="5-before-custom-card"
src="https://github.com/user-attachments/assets/c725f642-3d22-4b8d-8fb0-c702e415594d"
/> | <img width="748" height="897" alt="6-after-custom-card"
src="https://github.com/user-attachments/assets/afcbcb01-fed9-4ef3-87c2-883d73594edc"
/> |

Judges. System One on OpenRouter serves two of its protocols, so it
shows Protocol. The LLM judge on OpenAI serves one, so it shows "via
OpenAI Chat Completions".

| Before | After |
| --- | --- |
| <img width="748" height="504" alt="7-before-judges"
src="https://github.com/user-attachments/assets/a089a1bd-9f72-4ca6-bff1-e1a652a83ec1"
/> | <img width="748" height="503" alt="8-after-judges"
src="https://github.com/user-attachments/assets/61ef9ae3-e080-4e54-99a1-dc43d4960ba6"
/> |

The custom card at 390 px:

<img width="358" height="1153" alt="9-after-custom-card-390"
src="https://github.com/user-attachments/assets/dce35d02-44b1-4ccb-8362-c84129e1eb87"
/>

## Verified

The branch is based on `main` at 1cc37ac. I tested its tip, each suite
in a new rootless Podman container (`python:3.12`, `--init`, uv on PATH,
Node for the route suite):

- Backend suite (`python3 app/tests/parallel.py`): 2029 tests in 118
modules, OK. `main` at 1cc37ac, in the same container: 1992 tests in
116 modules, OK.
- Route suite (`python3 app/tests/parallel.py app/ui_tests`): 441 tests
in 42 modules, OK. `main` at 1cc37ac: 431 tests in 41 modules, OK.
- New modules: `test_provider_registry` (17 tests: the merge, `null`,
partial environment values, the old settings, record checks, the preset
host rule, keys never in a record), `test_judge_decisions` (18 tests:
each decision protocol's request and reply, `usage.cost`, `refusal`,
402, no redirect, the cached model list and its fallback),
`ui_tests.test_judge_cards` (2 tests: the protocol row and the live
list). Existing modules gained tests for the relay's headers, the
builder's provider filter, Harbor recipes on Anthropic and the Settings
cards.
- End to end, against the server (`app/app.py`) and the CLI, in a
container with no network, with a new `NULLIUS_HOME` for each case and
fixture keys only. 37 checks, all pass:
  - No settings: the four presets and the OpenAI-compatible card.
- A custom provider in `settings.json`: its card, its merged record, the
default path for a protocol without an entry, System One offers it for
its `decisions` endpoint, no key in the page.
- `MODEL_GW='{"base": "https://gw-eu.example/v1", "endpoints":
{"decisions": null}}'` in the environment: the base changes, the name
stays the file's, the endpoint goes, System One no longer offers it. The
card shows Base read-only and Name editable. The file does not change.
- `nullius migrate --providers`: the preview names the moves, prints no
value and leaves the file as it was. `--apply` writes the record,
removes the old lines, keeps the key and the file mode 0600. A second
run finds nothing to move.
- Old variables in the environment (`MODEL_API_URL`, `MODEL_<ID>_NAME`,
`MODEL_<ID>_API_URL`, `LLM_JUDGE_PROVIDER=local:<ID>`): read as before.
- System One's model list on OpenRouter, offline: the route answers with
the built-in list in less than the 5 s timeout, or with a cached list,
however old.
- Browser (Chromium, Playwright), on this branch and on `main`, at 1280
px and 390 px, light and dark, with no network: the provider list, a
preset card and a custom card with Endpoints open, and both judge cards.
No page errors and no horizontal scroll.

Not verified: live calls to OpenRouter Decisions and OpenAI Decisions,
and a whole run on Anthropic. The tests use fixtures.

## Risk

A wrong record or header can send a call to the wrong URL, or with the
wrong key header. Undo with a revert. The old settings are still read,
so a revert keeps working with them. Records written by a card stay in
`settings.json` and the older code ignores them.

- **Migration on edit.** Saving a custom provider's card moves
`MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` into `MODEL_<ID>`. After a
revert, that provider loses its name and base until they are set again.
`nullius migrate --providers` without `--apply` shows what would move.
- **New reserved names.** `OPENROUTER`, `OPENAI`, `ANTHROPIC`,
`TYPESAFE` and `MODEL_<ID>` are now settings. An agent key listed under
one of these names is left out, and Settings says so. A shell variable
such as `MODEL_PATH` that is not a JSON object is not read as a record.
- **`--provider anthropic` on the CLI.** The key header now follows the
agent's protocol. An agent that speaks only `chat` is refused on
Anthropic, where it used to get `x-api-key` on OpenAI-style paths.
- **Runs and custom endpoint paths.** The relay forwards the agent's own
path below the provider's base. An Endpoints row changes where the
judges and the test ask, not where a run's calls go.
- **`openai-decisions` field names** come from OpenAI's documentation,
not from a live reply.
- **Open PRs on the same files.** This change does not depend on any of
them. Whichever merges second rebases. A test merge of this branch with
each one:
- #124 (OpenRouter serving providers, `--served-by`): conflicts in
`providers.py`, `standard/run.py`, `standard/relay.py` and
`docs/settings/model-providers.md`. Each conflict is small. Both add
functions at the same place in `providers.py`. This branch removes
`run.PROVIDERS`, where #124 adds `routing` to the OpenRouter entry: that
flag becomes a field of the OpenRouter preset. Both change
`Relay._headers`.
- #133 (stacked on #124): the same four files, and also
`standard/fidelity.py` (one `key_header=` line, now `r.key_header`) and
`docs/standard.md` (one table row each).
- #134 (stacked on #133): the same as #133, and also
`ui_tests/test_settings_page.py` (one placeholder assertion, which both
change).
- #124, #133 and #134 also conflict with `main` alone in
`standard/merge.py` (and #133, #134 in `tests/test_standard_shard.py`),
which this branch does not touch.
Accept named models for the current recipe when only provider pins differ. Keep the full recipe key check for page output and hide cached models after a model change.

Exercise overlapping successful and failed compile answers in both orders.
# Conflicts:
#	app/nullius/providers.py
#	app/nullius/standard/relay.py
#	app/nullius/standard/run.py
#	docs/settings/model-providers.md
# Conflicts:
#	app/nullius/standard/recipe.py
Ask for serving providers when compilation names the model sent by each field. Share the five-minute cache with the picker and update each row in place.

Hide an unpinned row while its list is pending or contains one entry. Keep pinned rows and empty or failed lists visible. Ignore responses for models that have changed.

Cover visibility, request reuse and model changes under Node. Update the builder and provider guides.
The list of serving providers was already asked through the page's request helper, passed in under the name fetch. The check that every request of a page goes through the helper reads the scripts' text and took that name for a bare call. The parameter is now named request.
@aitianshi
aitianshi merged commit 5bcfd82 into ThinkFlowLab:main Nov 17, 2026
2 checks passed
aitianshi added a commit that referenced this pull request Nov 17, 2026
… work on a unit and cap its model calls (#191)

Closes #190

## What

A standard run can now force its limits, and the agent's time limit
covers its whole work on a unit.

- **Time limits by phase.** `run --timeout PHASE=SECONDS` forces the
limit of a phase: `setup`, `image`, `agent`, `verify` or `score`. Empty
keeps the default, which comes from the task, then the benchmark, then
Nullius. `config.json` records what was forced under
`standard.timeouts`.
- **One agent limit per unit.** The agent's limit is one deadline for
its whole work on a unit: every turn of a conversation, and the
simulated user's time between them. Before, it applied to each reply. A
capability keeps one limit per add and per recall, because a memory adds
a unit's sessions one by one. τ³-bench's `limits.turn_seconds` goes from
600 s a reply to 7200 s a conversation.
- **Turns.** `run --max-turns N` sets the turns of a conversation. The
builder offers it only on a benchmark that holds conversations.
- **A cap on model calls.** `run --max-calls N` caps the model calls an
agent makes on a unit. The relay counts each answered call on the
agent's route; rate-limited and failed calls are not counted, and
neither are the judge's or the simulator's. Past the cap the relay
refuses the call with a 402, and the unit ends as `UnitCallCapError`, an
agent error that is not scored. It is for agents only: a capability's
route also carries the benchmark's reader, so a capability run refuses
it.
- **A benchmark's own bounds.** They are now options marked `limit`.
τ³-bench gets Steps per conversation (`--max-steps`, from 1 up to its
default of 200) and Tool errors per conversation (`--max-errors`).
OSWorld gets Steps per task (each agent script's `--max_steps`); left
empty it uses the chosen agent's own value, declared with the new `by`
key.
- **Builder, Limits step.** It shows Scope, then Time limits in minutes,
then a section named after the benchmark when it has bounds of its own.
An empty field shows the value the run uses: in the text colour when the
benchmark or the agent declares it, grey when it is Nullius's default
(priority: Nullius < agent < benchmark). Nothing is typed into a field
for the user.

## Touches

- `app/nullius/standard/`: `run.py`, `relay.py`, `container.py`,
`job.py`, `cli.py`, `recipe.py`, `definitions.py`
- `app/nullius/runs/classify.py`: `UnitCallCapError` is an agent error
- Catalogue: τ³-bench and OSWorld's `standard.json`, their drivers and
`upstream-settings.json`
- `app/static/builder/`: the Limits step
- Docs: `docs/cli/run.md`, `docs/standard.md`, `docs/builder.md`, the
field guide and the generated supported-benchmark pages

## Before and after

The Limits step for τ³-bench, before and after:

| Before | After |
| --- | --- |
| <img width="440" alt="Limits step on main: Scope only, with tasks,
concurrency, attempts and two spending caps"
src="https://github.com/user-attachments/assets/80b468a4-326f-4057-a4b2-5551ea93e642"
/> | <img width="440" alt="Limits step on this branch: Scope with turns
and model calls, Time limits in minutes, then a tau3-bench section with
steps and tool errors"
src="https://github.com/user-attachments/assets/6278494c-6c9f-456f-b739-9578494315ee"
/> |

## Verified

- Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging
`main`, with #124's serving providers and #150's Boat VMs).
- New tests:
- A whole conversation that ends past the agent limit, though no single
turn does.
  - Four one-call turns cut on the fourth call by `max_calls=3`.
  - The relay counts only answered calls on the agent's route.
- A slow capability whose three adds each fit a 1 s limit while their
total does not.
  - A capability run refuses `--max-calls`.
- Recipe validation of the limits, including a value that rounds to 0 s.
  - τ³-bench steps above 200 are refused.
- OSWorld's `--max_steps` is replaced, or added when the agent's command
doesn't set it.
- Against the installed τ³-bench upstream: the driver uses 200 steps and
10 errors by default, and 7 and 3 when the run sets them.
- Checked in the browser, light and dark, for τ³-bench and OSWorld.

## Risk

- **Agent timeouts on conversation benchmarks.** A conversation that
used to fit 600 s per reply now has a single limit for the whole
conversation, 7200 s on τ³-bench. A very long conversation can now time
out where it didn't before. Undo by forcing a larger `--timeout
agent=SECONDS`, or by reverting `limits.turn_seconds`.
- **Comparability.** A run whose limits differ from the defaults is not
yet marked as not comparable with the leaderboard.
- **Overlap.** #174 (the agent's setup timed apart from its run) and
#133 (a benchmark's own limits) change the same parts of `run.py` and
the catalogue, so expect merge conflicts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: a run cannot choose, or see, which provider serves a model on OpenRouter

2 participants