Repository navigation
Closed
Feature
Copy link
Labels
P2Planned: limited in scope, has a way around it, or a missing featurePlanned: limited in scope, has a way around it, or a missing featureenhancementNew feature or requestNew feature or request
Description
Priority
extreme. The change reaches the relay's key headers and the settings that hold keys.
Components
- app/nullius (backend)
- app/web (routes)
- templates and static files
- docs and the site
- the standard (ports, kits, relay, runs)
Today
Each feature keeps its own list of model providers, and the lists disagree:
| Where | Providers |
|---|---|
app/nullius/providers.py (PROVIDERS, PROVIDER_BASE, PROVIDER_KEY) |
openrouter, openai, custom |
app/static/builder/command-builder-core.js (PROVIDERS) |
a copy of the above |
app/nullius/providers.py (TESTED) |
the above and anthropic |
app/nullius/standard/run.py (PROVIDERS), the CLI's --provider |
openai, openrouter, anthropic, custom |
app/nullius/judge/client.py (LLM_PROVIDERS) |
openrouter, openai, local, local:<ID> |
app/nullius/judge/client.py (SYSTEM_ONE_PROVIDERS) |
typesafe, openrouter |
app/nullius/onboarding.py (PROVIDER_KEYS) |
OpenRouter, OpenAI, MODEL_API_URL |
The effects:
- Anthropic is in Settings, the provider test, the CLI and the relay, but not in the builder, the LLM judge or onboarding. A user with only an Anthropic key is told that no provider is set up.
- The OpenAI-compatible endpoint is
customin recipes and the builder, andlocalorlocal:<ID>in Settings and the judge. - OpenRouter's base URL is written eight times, in five files.
- System One speaks only TypeSafe's wire format (
/systemone). It cannot ask OpenRouter Decisions (/api/alpha/decisions, models of many makers) or OpenAI Decisions (/v1/decisions). Its OpenRouter model list is fixed in code. It ignores theusage.costthat OpenRouter returns, and it drops arefusalanswer without a word. - A custom provider has a URL and a key only. It cannot say that it serves another protocol at another path.
Wanted
One definition of each model provider, which every feature reads. A feature asks for the protocol that it speaks and gets the providers that can serve it.
- Protocols. A protocol is a wire format, defined once: its default path and how the key is sent.
chat,responsesandmessages(the names agent definitions use),systemone,decisions(OpenRouter) andopenai-decisions. - One record shape. A provider is
{name, base, key, endpoints}.keynames the secret setting that holds the key, never the key.endpointsmaps a protocol to a path (appended tobase) or a full URL. A preset (OpenRouter, OpenAI, Anthropic, TypeSafe) is the same record with its name, base, key setting and known endpoints fixed. A custom provider is a record that the user writes whole. - Storage in layers. A record is one JSON setting:
MODEL_<ID>for a custom provider,OPENROUTER,OPENAI,ANTHROPICorTYPESAFEfor the additions to a preset. The preset in code,settings.jsonand the environment merge field by field. An environment value is a partial JSON object, andnullremoves a field (RFC 7396). Settings shows read-only only the fields that the environment sets. - Keys stay separate secret settings, with the names they have today, so masking, redaction and "a run gets only its provider's key" do not change.
- Migration.
MODEL_<ID>_NAMEandMODEL_<ID>_API_URLare still read asnameandbase. Saving a card moves them into the record.nullius migrate --providerspreviews the same rewrite, and--applywrites it. Recipes do not change.LLM_JUDGE_PROVIDERstill readslocalandlocal:<ID>. - Consumers filter by protocol. The builder, standard runs and the CLI ask for what the agent speaks. The LLM judge asks for
chatormessages. System One asks for a decision protocol. Onboarding counts any run provider. The duplicate lists go. - Settings cards. A card shows Base and, in a collapsed Endpoints section, one row for each protocol. A preset's known endpoints are fixed. An endpoint added to a preset is a path, or a full URL on the preset's own host, because the preset's key goes there.
- System One in each decision protocol, with one adapter for the TypeSafe shape (
systemone,decisions) and one foropenai-decisions. It keepsusage.costand recordsrefusalas no answer. - Judge cards. Each judge has Provider, then Protocol when the provider serves two that the judge speaks (else one quiet line), then Model. New settings
LLM_JUDGE_PROTOCOLandSYSTEM_ONE_PROTOCOL. - A live decision-model list. On OpenRouter, System One offers OpenRouter's public list of decision models, asked without a key, kept 6 hours, with the fixed list as a fallback.
Acceptance
-
app/nullius/providers.pyis the only place that names a provider's base URL and key setting. The builder, the provider test, standard runs, the CLI, both judges and onboarding read it. - A provider record never holds a key. Settings refuses a record whose
keynames another setting than its own. - A record merges preset, file and environment field by field. A partial environment object changes only its fields, and
nullremoves a field. Tests cover both. - Old
MODEL_<ID>_NAMEandMODEL_<ID>_API_URLsettings still work.nullius migrate --providerspreviews the rewrite and--applywrites it. Saved recipes open unchanged. - The builder offers Anthropic only to an agent of the standard that speaks
messages. Review and Run refuse a provider that does not serve what the recipe's agents and judge speak. - The relay sends
x-api-keyandanthropic-versiononly on amessagesroute, and a bearer header elsewhere. - An endpoint added to a preset is refused unless it is a path or a URL on the preset's host.
- System One asks
systemone,decisionsandopenai-decisions, keepsusage.cost, and records arefusalas no answer. Tests use fixtures, with no live calls. - The judge cards show Protocol only when the chosen provider serves two protocols of that judge.
-
docs/settings/model-providers.md,system-one.md,llm-judge.md, the settings reference and the AGENTS.md map describe the registry.
Dependencies and related
- Settings: a provider's card has no Remove after its first key is saved, until a reload #118 (closed by fix(settings): bring a provider card's Remove up to date after a save #125) changed the same provider cards: their Remove button. This work keeps that behaviour.
- Standard: a run cannot choose, or see, which provider serves a model on OpenRouter #123 and PR feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 add OpenRouter serving providers (
--served-by) to runs. PR feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control #133 and PR feat(catalogue): add OSWorld2 with OSWorld-V2's own agents, its services on the machine and a null control #134 are stacked on feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124. All three editapp/nullius/providers.py,app/nullius/standard/run.pyandrelay.py. A serving-provider pin fits as an option of the OpenRouter preset. Neither change depends on the other. The one that merges second rebases.
Metadata
Metadata
Assignees
Labels
P2Planned: limited in scope, has a way around it, or a missing featurePlanned: limited in scope, has a way around it, or a missing featureenhancementNew feature or requestNew feature or request
Activity
feat(settings): define model providers once, and pick them by protoco…