Skip to content

feat(settings): define model providers once, and pick them by protocol everywhere - #148

Merged
harenome merged 7 commits into
ThinkFlowLab:mainfrom
harenome:feat/provider-registry
Nov 16, 2026
Merged

harenome merged 7 commits into
ThinkFlowLab:mainfrom
harenome:feat/provider-registry

Conversation

@harenome

@harenome harenome commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #147

What

Each model provider is now one record that every feature reads. A feature asks for the protocol that it speaks, and gets the providers that serve it.

  • The registry (app/nullius/providers.py). PROTOCOLS defines each wire format once: its default path and how the key is sent (chat, responses, messages, systemone, decisions, openai-decisions). PRESETS holds OpenRouter, OpenAI, Anthropic and TypeSafe. record() returns {name, base, key, endpoints} for a preset or a custom provider. key is the name of the secret setting, never the key.
  • Storage (app/nullius/settings.py). A record is one JSON setting of a new kind, provider. A custom provider is MODEL_<ID>. The endpoints added to a preset are OPENROUTER, OPENAI, ANTHROPIC or TYPESAFE. For these settings the environment does not replace the file. Its value is a partial object, merged over the file's value (RFC 7396, null removes a field). Settings shows read-only only the fields that the environment sets. Keys keep their names and stay separate secrets.
  • Migration. MODEL_<ID>_NAME and MODEL_<ID>_API_URL are still read as name and base. Saving a card moves them into the record. nullius migrate --providers previews that rewrite of settings.json, and --apply writes it. Recipes do not change. local and local:<ID> still read as custom.
  • Consumers. The registry is read by the builder, the provider test, onboarding, execution, standard runs, the relay, the CLI's --provider and both judges. The builder's JavaScript copy of the list is gone. The builder offers a provider only where it serves what the recipe's agents and judge speak, so Anthropic appears for an agent of the standard that speaks messages. Review, Run (web/execute.py) and run.Run refuse a provider that does not fit. The relay sends x-api-key and anthropic-version on a messages route only, and a bearer header on a benchmark's own routes.
  • Settings cards. A card shows Base and a collapsed Endpoints section with one row for each protocol. The placeholder is the default URL. A base on a preset's host suggests that preset's endpoints. A preset's known endpoints are fixed. An endpoint added to a preset is a path, or a full URL on the preset's own host, because the preset's key goes there.
  • System One (app/nullius/judge/client.py) asks each decision protocol: the TypeSafe shape on systemone and decisions, and its own adapter for openai-decisions. It keeps the provider's usage.cost and records a refusal as no answer. A 402 says what it means. A judge never follows a redirect with the key.
  • Judge cards. Each judge has Provider, then Protocol when the provider serves two protocols of that judge (else one quiet line, "via ..."), then Model. Each judge's provider is a select whatever the count, because the list is the user's own providers and grows or shrinks with them; cards are kept for fixed sets of choices. New settings: LLM_JUDGE_PROTOCOL, SYSTEM_ONE_PROTOCOL. On OpenRouter, System One offers OpenRouter's public list of decision models, asked without a key, cached 6 hours, with the fixed list as the fallback.

The seven commits are atomic and in reading order: registry and storage, consumers, Settings cards, decision protocols, judge cards, tests, docs.

Touches

backend (providers, settings, migrate, onboarding, execution, judge/client, recipes/launch) · the standard (run, relay, recipe, cli, definitions, fidelity) · routes (web/settings, web/execute, web/filters) · templates and static (Settings cards, judge cards, builder core) · docs (docs/settings/model-providers.md, system-one.md, llm-judge.md, README.md, agent-keys.md, docs/builder.md, docs/judgement.md, docs/standard.md, docs/cli/README.md, AGENTS.md, app/.env.example)

Before and after

Settings at 1280 px, light theme. Fixture keys only.

Model providers. TypeSafe is a preset like the others, so it comes after the run providers.

Before After
1-before-providers 2-after-providers

A preset card (OpenRouter), Endpoints open. Known endpoints are fixed. Other protocols are editable rows.

Before After
3-before-preset-card 4-after-preset-card

A custom provider, Endpoints open, with one endpoint set.

Before After
5-before-custom-card 6-after-custom-card

Judges. System One on OpenRouter serves two of its protocols, so it shows Protocol. The LLM judge on OpenAI serves one, so it shows "via OpenAI Chat Completions".

Before After
7-before-judges 8-after-judges

The custom card at 390 px:

9-after-custom-card-390

Verified

The branch is based on main at 1cc37ac. I tested its tip, each suite in a new rootless Podman container (python:3.12, --init, uv on PATH, Node for the route suite):

  • Backend suite (python3 app/tests/parallel.py): 2029 tests in 118 modules, OK. main at 1cc37ac, in the same container: 1992 tests in 116 modules, OK.
  • Route suite (python3 app/tests/parallel.py app/ui_tests): 441 tests in 42 modules, OK. main at 1cc37ac: 431 tests in 41 modules, OK.
  • New modules: test_provider_registry (17 tests: the merge, null, partial environment values, the old settings, record checks, the preset host rule, keys never in a record), test_judge_decisions (18 tests: each decision protocol's request and reply, usage.cost, refusal, 402, no redirect, the cached model list and its fallback), ui_tests.test_judge_cards (2 tests: the protocol row and the live list). Existing modules gained tests for the relay's headers, the builder's provider filter, Harbor recipes on Anthropic and the Settings cards.
  • End to end, against the server (app/app.py) and the CLI, in a container with no network, with a new NULLIUS_HOME for each case and fixture keys only. 37 checks, all pass:
    • No settings: the four presets and the OpenAI-compatible card.
    • A custom provider in settings.json: its card, its merged record, the default path for a protocol without an entry, System One offers it for its decisions endpoint, no key in the page.
    • MODEL_GW='{"base": "https://gw-eu.example/v1", "endpoints": {"decisions": null}}' in the environment: the base changes, the name stays the file's, the endpoint goes, System One no longer offers it. The card shows Base read-only and Name editable. The file does not change.
    • nullius migrate --providers: the preview names the moves, prints no value and leaves the file as it was. --apply writes the record, removes the old lines, keeps the key and the file mode 0600. A second run finds nothing to move.
    • Old variables in the environment (MODEL_API_URL, MODEL_<ID>_NAME, MODEL_<ID>_API_URL, LLM_JUDGE_PROVIDER=local:<ID>): read as before.
    • System One's model list on OpenRouter, offline: the route answers with the built-in list in less than the 5 s timeout, or with a cached list, however old.
  • Browser (Chromium, Playwright), on this branch and on main, at 1280 px and 390 px, light and dark, with no network: the provider list, a preset card and a custom card with Endpoints open, and both judge cards. No page errors and no horizontal scroll.

Not verified: live calls to OpenRouter Decisions and OpenAI Decisions, and a whole run on Anthropic. The tests use fixtures.

Risk

A wrong record or header can send a call to the wrong URL, or with the wrong key header. Undo with a revert. The old settings are still read, so a revert keeps working with them. Records written by a card stay in settings.json and the older code ignores them.

Add the provider registry to nullius.providers: the protocols (wire
formats with a default path and an authentication), the presets
(OpenRouter, OpenAI, Anthropic, TypeSafe) and one record shape for
presets and custom providers. A record names its key setting, never
the key. An endpoint is a path appended to the base with one slash, or
a full URL that passes the checks a base gets.

Store a record as one JSON setting: MODEL_<ID> for a custom provider,
the preset's name for a preset's additions. A new settings kind,
provider, merges the environment's partial object over the file's
(RFC 7396, null removes a field) and reports which fields the
environment sets. The old MODEL_<ID>_NAME and MODEL_<ID>_API_URL
settings stay readable; the record wins where both are set.

nullius migrate --providers previews, and with --apply writes, the
rewrite of the old names in settings.json. Nothing reads the registry
yet.
Each consumer now asks the registry for the protocol it speaks and
drops its own provider list:

- The builder gets the run providers from the backend (schema
  providers) instead of a JavaScript copy, and offers each one only
  where its protocols fit the recipe's agents. Anthropic is a run
  provider for agents that speak messages; it is not offered to an
  agent that speaks only chat.
- Standard runs, the CLI's --provider and fidelity read base and key
  from the record. The relay sends the key as the protocol of the
  agent says (x-api-key for messages, else a bearer header).
- A run on a custom provider reads its base from the record.
- The LLM judge offers every provider that may serve chat or messages,
  with a small Messages adapter for Anthropic. System One offers every
  provider that lists systemone; the OpenRouter list gains
  typesafe/jev-1.14.
- Onboarding counts any provider with a key or a base, so an
  Anthropic-only user is set up.
- custom is the one name of the OpenAI-compatible kind for the judge;
  local and local:<id> are still read.

Help texts no longer repeat base URLs.
Build the Model providers cards from the registry: presets and custom
providers are one list, and custom is the card name of the
OpenAI-compatible kind. A card keeps Name, Base and Key; a preset shows
its base read-only. A collapsed Endpoints section has one row per
protocol: the placeholder is the default URL, a tooltip names the
format, and a base on a known host suggests the endpoint. A preset's
own endpoints are read-only; the user can add other protocols.

A field the environment sets is read-only; the other fields stay
editable. A save writes the record's file layer whole and moves a
custom provider's old name and URL lines into its record. "+ Add"
starts from a preset or from a blank custom provider, which is now
saved as a record.
System One now asks any provider that lists a decision protocol:

- systemone, as before. The reply's usage.cost is kept when the
  provider gives it, a refusal answer is recorded in refusals instead
  of being dropped, and a 402 says that the key's credit is spent or
  the model is not free.
- decisions (OpenRouter's alpha API): the same request and reply
  shape, with a confidence on yes/no answers and errors as
  {error: {code, message}}. On OpenRouter, TypeSafe's models stay on
  systemone and other makers' models go to decisions.
- openai-decisions: its own adapter. The state goes as input, the
  questions as an array by name (a yes/no question is a predicate,
  its criteria folded into its instructions), and the array of answers
  is read back into the record.

The judgement page names the model that answered instead of TypeSafe,
drops the "not calibrated" contrast and counts refused questions.
…ocol

A judge card reads Provider, then Protocol when the chosen provider lists
two that the judge speaks, then Model. The provider is the shared select.
With one protocol, a muted line under it names the format the judge uses.
The protocol select is saved only once a protocol is chosen, so a provider
change never writes one.
…pters

New tests, with fakes only and no request sent:

- the registry: preset fields fixed, the three layers merged field by
  field, endpoints per protocol, null removal, the environment's
  partial record locking only its fields, path and URL joining and
  their checks, the old settings read and overridden, a shell's
  MODEL_NAME not taken for a provider, nullius migrate --providers;
- the consumers: protocol filtering (Anthropic only for messages),
  the relay's key header from the protocol, an Anthropic-only user
  counted as set up, and no key in a record, the builder's providers
  or its payload;
- System One on systemone, decisions and openai-decisions (cost,
  refusal, 402, error shape, request and reply shapes), and the LLM
  judge on Anthropic's Messages;
- the Settings page: read-only environment fields, Endpoints rows
  saved into the record, a preset's own endpoints fixed.

Existing tests follow the custom name and the record setting.
Document the providers and the protocols each one serves, which
protocol each feature asks for, the custom provider's Endpoints rows,
the provider record in settings.json and its environment override,
how the old settings are read and migrated, System One's decision
protocols and the LLM judge on Anthropic. Update the AGENTS.md map
line of nullius.providers.
@harenome
harenome merged commit ce8b39d into ThinkFlowLab:main Nov 16, 2026
2 checks passed
@harenome
harenome deleted the feat/provider-registry branch November 16, 2026 13:59
sirotaoleg added a commit to sirotaoleg/nullius that referenced this pull request Nov 16, 2026
The new main beneath it: OSWorld2's tests take their relay's key header from run.key_header, since ThinkFlowLab#148 removed run.PROVIDERS and Run.unit_relay now reads Run.key_header.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Settings: define each model provider once, and let every feature pick providers by protocol

1 participant