Repository navigation
feat(settings): define model providers once, and pick them by protocol everywhere - #148
Merged
Merged
Conversation
Add the provider registry to nullius.providers: the protocols (wire formats with a default path and an authentication), the presets (OpenRouter, OpenAI, Anthropic, TypeSafe) and one record shape for presets and custom providers. A record names its key setting, never the key. An endpoint is a path appended to the base with one slash, or a full URL that passes the checks a base gets. Store a record as one JSON setting: MODEL_<ID> for a custom provider, the preset's name for a preset's additions. A new settings kind, provider, merges the environment's partial object over the file's (RFC 7396, null removes a field) and reports which fields the environment sets. The old MODEL_<ID>_NAME and MODEL_<ID>_API_URL settings stay readable; the record wins where both are set. nullius migrate --providers previews, and with --apply writes, the rewrite of the old names in settings.json. Nothing reads the registry yet.
Each consumer now asks the registry for the protocol it speaks and drops its own provider list: - The builder gets the run providers from the backend (schema providers) instead of a JavaScript copy, and offers each one only where its protocols fit the recipe's agents. Anthropic is a run provider for agents that speak messages; it is not offered to an agent that speaks only chat. - Standard runs, the CLI's --provider and fidelity read base and key from the record. The relay sends the key as the protocol of the agent says (x-api-key for messages, else a bearer header). - A run on a custom provider reads its base from the record. - The LLM judge offers every provider that may serve chat or messages, with a small Messages adapter for Anthropic. System One offers every provider that lists systemone; the OpenRouter list gains typesafe/jev-1.14. - Onboarding counts any provider with a key or a base, so an Anthropic-only user is set up. - custom is the one name of the OpenAI-compatible kind for the judge; local and local:<id> are still read. Help texts no longer repeat base URLs.
Build the Model providers cards from the registry: presets and custom providers are one list, and custom is the card name of the OpenAI-compatible kind. A card keeps Name, Base and Key; a preset shows its base read-only. A collapsed Endpoints section has one row per protocol: the placeholder is the default URL, a tooltip names the format, and a base on a known host suggests the endpoint. A preset's own endpoints are read-only; the user can add other protocols. A field the environment sets is read-only; the other fields stay editable. A save writes the record's file layer whole and moves a custom provider's old name and URL lines into its record. "+ Add" starts from a preset or from a blank custom provider, which is now saved as a record.
System One now asks any provider that lists a decision protocol:
- systemone, as before. The reply's usage.cost is kept when the
provider gives it, a refusal answer is recorded in refusals instead
of being dropped, and a 402 says that the key's credit is spent or
the model is not free.
- decisions (OpenRouter's alpha API): the same request and reply
shape, with a confidence on yes/no answers and errors as
{error: {code, message}}. On OpenRouter, TypeSafe's models stay on
systemone and other makers' models go to decisions.
- openai-decisions: its own adapter. The state goes as input, the
questions as an array by name (a yes/no question is a predicate,
its criteria folded into its instructions), and the array of answers
is read back into the record.
The judgement page names the model that answered instead of TypeSafe,
drops the "not calibrated" contrast and counts refused questions.
…ocol A judge card reads Provider, then Protocol when the chosen provider lists two that the judge speaks, then Model. The provider is the shared select. With one protocol, a muted line under it names the format the judge uses. The protocol select is saved only once a protocol is chosen, so a provider change never writes one.
…pters New tests, with fakes only and no request sent: - the registry: preset fields fixed, the three layers merged field by field, endpoints per protocol, null removal, the environment's partial record locking only its fields, path and URL joining and their checks, the old settings read and overridden, a shell's MODEL_NAME not taken for a provider, nullius migrate --providers; - the consumers: protocol filtering (Anthropic only for messages), the relay's key header from the protocol, an Anthropic-only user counted as set up, and no key in a record, the builder's providers or its payload; - System One on systemone, decisions and openai-decisions (cost, refusal, 402, error shape, request and reply shapes), and the LLM judge on Anthropic's Messages; - the Settings page: read-only environment fields, Endpoints rows saved into the record, a preset's own endpoints fixed. Existing tests follow the custom name and the record setting.
Document the providers and the protocols each one serves, which protocol each feature asks for, the custom provider's Endpoints rows, the provider record in settings.json and its environment override, how the old settings are read and migrated, System One's decision protocols and the LLM judge on Anthropic. Update the AGENTS.md map line of nullius.providers.
sirotaoleg
added a commit
to sirotaoleg/nullius
that referenced
this pull request
Nov 16, 2026
The new main beneath it: OSWorld2's tests take their relay's key header from run.key_header, since ThinkFlowLab#148 removed run.PROVIDERS and Run.unit_relay now reads Run.key_header.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #147
What
Each model provider is now one record that every feature reads. A feature asks for the protocol that it speaks, and gets the providers that serve it.
app/nullius/providers.py).PROTOCOLSdefines each wire format once: its default path and how the key is sent (chat,responses,messages,systemone,decisions,openai-decisions).PRESETSholds OpenRouter, OpenAI, Anthropic and TypeSafe.record()returns{name, base, key, endpoints}for a preset or a custom provider.keyis the name of the secret setting, never the key.app/nullius/settings.py). A record is one JSON setting of a new kind,provider. A custom provider isMODEL_<ID>. The endpoints added to a preset areOPENROUTER,OPENAI,ANTHROPICorTYPESAFE. For these settings the environment does not replace the file. Its value is a partial object, merged over the file's value (RFC 7396,nullremoves a field). Settings shows read-only only the fields that the environment sets. Keys keep their names and stay separate secrets.MODEL_<ID>_NAMEandMODEL_<ID>_API_URLare still read asnameandbase. Saving a card moves them into the record.nullius migrate --providerspreviews that rewrite ofsettings.json, and--applywrites it. Recipes do not change.localandlocal:<ID>still read ascustom.execution, standard runs, the relay, the CLI's--providerand both judges. The builder's JavaScript copy of the list is gone. The builder offers a provider only where it serves what the recipe's agents and judge speak, so Anthropic appears for an agent of the standard that speaksmessages. Review, Run (web/execute.py) andrun.Runrefuse a provider that does not fit. The relay sendsx-api-keyandanthropic-versionon amessagesroute only, and a bearer header on a benchmark's own routes.app/nullius/judge/client.py) asks each decision protocol: the TypeSafe shape onsystemoneanddecisions, and its own adapter foropenai-decisions. It keeps the provider'susage.costand records arefusalas no answer. A 402 says what it means. A judge never follows a redirect with the key.LLM_JUDGE_PROTOCOL,SYSTEM_ONE_PROTOCOL. On OpenRouter, System One offers OpenRouter's public list of decision models, asked without a key, cached 6 hours, with the fixed list as the fallback.The seven commits are atomic and in reading order: registry and storage, consumers, Settings cards, decision protocols, judge cards, tests, docs.
Touches
backend (
providers,settings,migrate,onboarding,execution,judge/client,recipes/launch) · the standard (run,relay,recipe,cli,definitions,fidelity) · routes (web/settings,web/execute,web/filters) · templates and static (Settings cards, judge cards, builder core) · docs (docs/settings/model-providers.md,system-one.md,llm-judge.md,README.md,agent-keys.md,docs/builder.md,docs/judgement.md,docs/standard.md,docs/cli/README.md,AGENTS.md,app/.env.example)Before and after
Settings at 1280 px, light theme. Fixture keys only.
Model providers. TypeSafe is a preset like the others, so it comes after the run providers.
A preset card (OpenRouter), Endpoints open. Known endpoints are fixed. Other protocols are editable rows.
A custom provider, Endpoints open, with one endpoint set.
Judges. System One on OpenRouter serves two of its protocols, so it shows Protocol. The LLM judge on OpenAI serves one, so it shows "via OpenAI Chat Completions".
The custom card at 390 px:
Verified
The branch is based on
mainat 1cc37ac. I tested its tip, each suite in a new rootless Podman container (python:3.12,--init, uv on PATH, Node for the route suite):python3 app/tests/parallel.py): 2029 tests in 118 modules, OK.mainat 1cc37ac, in the same container: 1992 tests in 116 modules, OK.python3 app/tests/parallel.py app/ui_tests): 441 tests in 42 modules, OK.mainat 1cc37ac: 431 tests in 41 modules, OK.test_provider_registry(17 tests: the merge,null, partial environment values, the old settings, record checks, the preset host rule, keys never in a record),test_judge_decisions(18 tests: each decision protocol's request and reply,usage.cost,refusal, 402, no redirect, the cached model list and its fallback),ui_tests.test_judge_cards(2 tests: the protocol row and the live list). Existing modules gained tests for the relay's headers, the builder's provider filter, Harbor recipes on Anthropic and the Settings cards.app/app.py) and the CLI, in a container with no network, with a newNULLIUS_HOMEfor each case and fixture keys only. 37 checks, all pass:settings.json: its card, its merged record, the default path for a protocol without an entry, System One offers it for itsdecisionsendpoint, no key in the page.MODEL_GW='{"base": "https://gw-eu.example/v1", "endpoints": {"decisions": null}}'in the environment: the base changes, the name stays the file's, the endpoint goes, System One no longer offers it. The card shows Base read-only and Name editable. The file does not change.nullius migrate --providers: the preview names the moves, prints no value and leaves the file as it was.--applywrites the record, removes the old lines, keeps the key and the file mode 0600. A second run finds nothing to move.MODEL_API_URL,MODEL_<ID>_NAME,MODEL_<ID>_API_URL,LLM_JUDGE_PROVIDER=local:<ID>): read as before.main, at 1280 px and 390 px, light and dark, with no network: the provider list, a preset card and a custom card with Endpoints open, and both judge cards. No page errors and no horizontal scroll.Not verified: live calls to OpenRouter Decisions and OpenAI Decisions, and a whole run on Anthropic. The tests use fixtures.
Risk
A wrong record or header can send a call to the wrong URL, or with the wrong key header. Undo with a revert. The old settings are still read, so a revert keeps working with them. Records written by a card stay in
settings.jsonand the older code ignores them.MODEL_<ID>_NAMEandMODEL_<ID>_API_URLintoMODEL_<ID>. After a revert, that provider loses its name and base until they are set again.nullius migrate --providerswithout--applyshows what would move.OPENROUTER,OPENAI,ANTHROPIC,TYPESAFEandMODEL_<ID>are now settings. An agent key listed under one of these names is left out, and Settings says so. A shell variable such asMODEL_PATHthat is not a JSON object is not read as a record.--provider anthropicon the CLI. The key header now follows the agent's protocol. An agent that speaks onlychatis refused on Anthropic, where it used to getx-api-keyon OpenAI-style paths.openai-decisionsfield names come from OpenAI's documentation, not from a live reply.--served-by): conflicts inproviders.py,standard/run.py,standard/relay.pyanddocs/settings/model-providers.md. Each conflict is small. Both add functions at the same place inproviders.py. This branch removesrun.PROVIDERS, where feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 addsroutingto the OpenRouter entry: that flag becomes a field of the OpenRouter preset. Both changeRelay._headers.standard/fidelity.py(onekey_header=line, nowr.key_header) anddocs/standard.md(one table row each).ui_tests/test_settings_page.py(one placeholder assertion, which both change).mainalone instandard/merge.py(and feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control #133, feat(catalogue): add OSWorld2 with OSWorld-V2's own agents, its services on the machine and a null control #134 intests/test_standard_shard.py), which this branch does not touch.