dan-petty · Nov 5, 2026 · Nov 5, 2026 · Nov 5, 2026
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-11-05
@@ -0,0 +1,31 @@
+# Proposal
+
+## Why
+
+Issue #3656: a call to a hosted gateway group (`claude-sonnet-5-5`, `claude-haiku-5-5`, `deepseek-v4.1-flash-1017`) is recorded as a local call at $0, and its tokens count as local savings. The ledger of the owner holds 354 such rows. The hosted groups set no `api_base`, so the gateway reply names no backend, and `is_local` then decides from the server that the call was sent to. That server is the gateway, a local host. The reply does name the deployment model (`x-litellm-model-name`), and `is_local` does not read it.
+
+A second gap: genai-prices holds no price for `deepseek-v4.1-flash-1017`. The warning for it does not say how to set a price.
+
+## Feasibility
+
+- The local ledger (`.data/ai/spend.db`, 2026-11-05) groups the rows with a deployment model that starts with `anthropic/` or `openrouter/`: 354 rows, `served_by` NULL in all, `pricing_source` `local` in all. Every row with a `served_by` is a cluster backend. The query is in issue #3656.
+- `genai_prices.calc_price` (genai-prices 0.1.13) prices `claude-sonnet-5-5` and `claude-haiku-5-5` at the provider `anthropic` (2.0 and 10.0 USD per million tokens for Sonnet, the prices of the gateway group), and `gemini-3.8-flash` at the provider `google`. It raises `LookupError` for `deepseek-v4.1-flash-1017` under the providers `litellm` and `openrouter`.
+- The upstream data (`https://raw.githubusercontent.com/pydantic/genai-prices/refs/heads/main/prices/new_data/v2/data.json`, 2026-11-05) lists `deepseek/deepseek-v4.1-flash` under `openrouter` and no `-1017` snapshot. Its price (0.0352 to 0.0448 USD input, 0.08 USD output per million) is not the price of the group (0.05 and 0.192 in `k8s/llm/gateway/config.yaml`), so a lookup by the base name would price the group wrongly.
+- `devops ai gateway render` and `tests/ai/benchmark/test_k8s_llm_gateway.py` already treat a deployment with no `api_base` as hosted.
+
+## Existing Tool Considered
+
+genai-prices prices the Claude and Gemini groups once the ledger asks it. For the DeepSeek group the open mechanism is the price override that the repository already has (`devops ai cost set-price`), keyed by the deployment model. The change builds no price table and no model-name matching. It makes the warning name the command. It deletes no code.
+
+## What Changes
+
+- `is_local` takes `served_model`. A reply of the `gateway` provider that names a deployment model and no backend is hosted. `calculate_request_cost` passes the deployment model, and the lifetime report groups the served usage by it, so a hosted group no longer counts in local savings or sets `local` on its tokens.
+- The warning for an unpriced hosted model adds the command `devops ai cost set-price <model> <prompt_rate> <completion_rate>`.
+- `tests/ai/benchmark/test_k8s_llm_gateway.py` fails when an in-cluster deployment sets no `api_base`: the ledger would count its calls as hosted.
+- The spec delta adds two requirements to `ai-spend-pricing`.
+- The change fits one pull request.
+
+## Impact
+
+- Rows that the ledger recorded before this change keep the source `local`. No migration: a row records what the process knew when it wrote it.
+- The DeepSeek group has the source `unknown` and a $0 cost until a person sets a price with the command that the warning names. The three known prices differ (the gateway config, the OpenRouter list, genai-prices), so the change chooses none.
@@ -0,0 +1,27 @@
+## ADDED Requirements
+
+### Requirement: A gateway reply that names a deployment model and no backend is hosted
+A reply of the `gateway` provider that names a deployment model in the `x-litellm-model-name` header and names no backend in the `x-litellm-model-api-base` header SHALL count as a hosted call, whatever host the gateway has. A reply that names a backend SHALL stay decided by that backend. The lifetime report SHALL apply the same rule to the rows of the ledger, and a row of a hosted call SHALL NOT count in local savings.
+
+#### Scenario: Hosted groups behind a loopback gateway
+- **WHEN** replies to `claude-haiku-5-5`, `claude-sonnet-5-5` and `gemini-3.8-flash` through a gateway on `127.0.0.1` name `anthropic/claude-haiku-5-5`, `anthropic/claude-sonnet-5-5` and `gemini/gemini-3.8-flash` and no backend
+- **THEN** the pricing sources are `genai_prices:anthropic`, `genai_prices:anthropic` and `genai_prices:google`, and each cost is more than $0
+
+#### Scenario: A reply that names a local backend
+- **WHEN** a reply names a deployment model and a cluster Ollama as its backend
+- **THEN** the call is local
+
+#### Scenario: The report counts a hosted group as hosted
+- **WHEN** the ledger holds one call that names `anthropic/claude-haiku-5-5` and no backend, and one call that names neither, both sent to `localhost:4000` through the `gateway` provider
+- **THEN** the report counts one local request
+
+### Requirement: The warning for an unpriced hosted model names the command that sets a price
+The warning that names a hosted model that neither genai-prices nor a price override prices SHALL give the command `devops ai cost set-price <model> <prompt_rate> <completion_rate>` with the model of the warning.
+
+#### Scenario: The DeepSeek group has no price
+- **WHEN** two replies name `openrouter/deepseek/deepseek-v4.1-flash-1017:floor` and no backend, and no price override exists
+- **THEN** both rows have the source `unknown` and a cost of $0, and one warning names the command `devops ai cost set-price`
+
+#### Scenario: A price override of the deployment model
+- **WHEN** a price override of `deepseek-v4.1-flash-1017:floor` is 0.05 USD input and 0.192 USD output per million tokens
+- **THEN** a call of 1,000,000 prompt and 1,000,000 completion tokens costs 0.242 USD with the source `custom_override`
@@ -0,0 +1,18 @@
+# Tasks
+
+## 1. Count a hosted gateway group as hosted
+
+Kind: behavior.
+
+Reads: `src/devops_cli/ai/spend/pricing.py` (`is_local`, `calculate_request_cost`), `src/devops_cli/ai/spend/ledger.py` (`_QUERY_SERVED_USAGE`, `_tally_and_annotate_servers`), `src/devops_cli/lang/en/messages.py` (`hosted_model_unpriced`). Changes: the same files, `tests/ai/spend/test_ai_spend_pricing.py`, `tests/ai/spend/test_ai_spend_ledger.py` and `tests/ai/benchmark/test_k8s_llm_gateway.py`.
+
+- [x] 1.1 Add the failing tests: `is_local` of a reply with a deployment model and no backend, the price of the Claude and Gemini groups behind a loopback gateway, the warning and the override of the DeepSeek group, the lifetime report, and the gateway config check that each local deployment sets `api_base`
+- [x] 1.2 Make `is_local` read the deployment model, pass it from `calculate_request_cost` and the report, and name the command in the warning
+  - Verification: `uv run devops ci test tests/ai/spend tests/ai/benchmark/test_k8s_llm_gateway.py`
+- [x] 1.3 Regenerate the documentation with `uv run devops docs generate --sync-readme`. No generated document holds the message, so no file changes
+  - Verification: `uv run devops ci docs`
+
+## Workflow follow-up
+
+- Pending a person: set the price of the group `deepseek-v4.1-flash-1017` with `devops ai cost set-price`.
+- Pending a person: review the diff in tuicr (tuicr -r origin/main..HEAD).
@@ -158,3 +158,29 @@ When a chat call that returns one reply fails after its status line, because of
 #### Scenario: An error frame after a ping
 - **WHEN** a chat call gets a stream with a `: ping` comment and then an error frame with `code` `"500"`
 - **THEN** the spend ledger has one row with the request type `chat_dispatch` and finish reason `error`
+
+### Requirement: A gateway reply that names a deployment model and no backend is hosted
+A reply of the `gateway` provider that names a deployment model in the `x-litellm-model-name` header and names no backend in the `x-litellm-model-api-base` header SHALL count as a hosted call, whatever host the gateway has. A reply that names a backend SHALL stay decided by that backend. The lifetime report SHALL apply the same rule to the rows of the ledger, and a row of a hosted call SHALL NOT count in local savings.
+
+#### Scenario: Hosted groups behind a loopback gateway
+- **WHEN** replies to `claude-haiku-5-5`, `claude-sonnet-5-5` and `gemini-3.8-flash` through a gateway on `127.0.0.1` name `anthropic/claude-haiku-5-5`, `anthropic/claude-sonnet-5-5` and `gemini/gemini-3.8-flash` and no backend
+- **THEN** the pricing sources are `genai_prices:anthropic`, `genai_prices:anthropic` and `genai_prices:google`, and each cost is more than $0
+
+#### Scenario: A reply that names a local backend
+- **WHEN** a reply names a deployment model and a cluster Ollama as its backend
+- **THEN** the call is local
+
+#### Scenario: The report counts a hosted group as hosted
+- **WHEN** the ledger holds one call that names `anthropic/claude-haiku-5-5` and no backend, and one call that names neither, both sent to `localhost:4000` through the `gateway` provider
+- **THEN** the report counts one local request
+
+### Requirement: The warning for an unpriced hosted model names the command that sets a price
+The warning that names a hosted model that neither genai-prices nor a price override prices SHALL give the command `devops ai cost set-price <model> <prompt_rate> <completion_rate>` with the model of the warning.
+
+#### Scenario: The DeepSeek group has no price
+- **WHEN** two replies name `openrouter/deepseek/deepseek-v4.1-flash-1017:floor` and no backend, and no price override exists
+- **THEN** both rows have the source `unknown` and a cost of $0, and one warning names the command `devops ai cost set-price`
+
+#### Scenario: A price override of the deployment model
+- **WHEN** a price override of `deepseek-v4.1-flash-1017:floor` is 0.05 USD input and 0.192 USD output per million tokens
+- **THEN** a call of 1,000,000 prompt and 1,000,000 completion tokens costs 0.242 USD with the source `custom_override`
@@ -146,19 +146,21 @@
 """
 
 # The backend that served a call decides whether it ran locally, so local usage is tallied by
-# server, provider and serving backend (#1631).
+# server, provider and serving backend (#1631). A gateway reply that names a deployment model and
+# no backend is hosted, so the tally also keeps the deployment model (#3656).
 _QUERY_SERVED_USAGE = """
 SELECT
     server,
     provider,
     served_by,
+    served_model,
     COUNT(*) as req_count,
     COALESCE(SUM(prompt_tokens), 0) as p_tokens,
     COALESCE(SUM(completion_tokens), 0) as c_tokens,
     COALESCE(SUM(CASE WHEN request_type = :embedding THEN prompt_tokens ELSE 0 END), 0) as e_tokens
 FROM ai_spend_records
 WHERE (:cutoff IS NULL OR timestamp >= :cutoff)
-GROUP BY server, provider, served_by;
+GROUP BY server, provider, served_by, served_model;
 """
 
 _QUERY_MODEL_BREAKDOWN = """
@@ -319,6 +321,7 @@ class _ServedUsage(NamedTuple):
     server: str
     provider: str
     served_by: str | None
+    served_model: str | None
     usage: _Usage
 
 
@@ -339,7 +342,10 @@ def _tally_and_annotate_servers(
     hosted: set[tuple[str, str]] = set()
     for group in served:
         if group.server != CONST_RESPONSE_CACHE_SERVER and is_local(
-            server=group.server, provider=group.provider, served_by=group.served_by
+            server=group.server,
+            provider=group.provider,
+            served_by=group.served_by,
+            served_model=group.served_model,
         ):
             local.append(group.usage)
         else:
@@ -663,6 +669,7 @@ def _query_served_usage(
                 server=r["server"],
                 provider=r["provider"],
                 served_by=r["served_by"],
+                served_model=r["served_model"],
                 usage=_Usage(
                     requests=int(r["req_count"] or 0),
                     prompt_tokens=int(r["p_tokens"] or 0),

@@ -15,6 +15,7 @@
 
 from devops_cli.ai.spend.models import ModelPricing
 from devops_cli.config.constants import (
+    CONST_AI_GATEWAY_PROVIDER,
     CONST_GENAI_PRICES_LITELLM_PROVIDER,
     CONST_LOCAL_PROVIDER_NAMES,
     CONST_LOCAL_TRANSPORT_LABELS,
@@ -274,26 +275,34 @@ def _is_local_gateway_configured(settings: _SettingsOnce) -> bool:
 
 
 def is_local(
-    server: str | None = None, provider: str | None = None, served_by: str | None = None
+    server: str | None = None,
+    provider: str | None = None,
+    served_by: str | None = None,
+    served_model: str | None = None,
 ) -> bool:
     """Return True if the call ran on local hardware.
 
     `served_by` is the backend that served the call, as the LLM gateway names it in its reply
     (`ai/client/network.py` reads the header). When the reply names it, it decides: a hosted route
     behind a local gateway is hosted, and a local backend behind a gateway on a public host is
-    local (#1631). LiteLLM names the API base of a route only when it knows it, so a gateway
-    route sets `api_base`. When the reply names no backend, the provider and the server the call
-    was sent to decide. The call loads the settings at most once, and only for a host that its
-    address and its name do not decide (#1420).
+    local (#1631). LiteLLM names the API base of a route only when it knows it, so a local route
+    sets `api_base` and a hosted group sets none. So a gateway reply that names a deployment
+    model, `served_model`, and no backend is hosted, whatever host the gateway has (#3656). When
+    the reply names neither, the provider and the server the call was sent to decide. The call
+    loads the settings at most once, and only for a host that its address and its name do not
+    decide (#1420).
     """
     settings = _SettingsOnce()
     if served_by and served_by.strip():
         return _is_local_server(served_by, settings)
+    is_gateway = provider is not None and provider.strip().lower() == CONST_AI_GATEWAY_PROVIDER
+    if is_gateway and served_model and served_model.strip():
+        return False
     if _is_local_provider(provider):
         return True
     if _is_local_server(server, settings):
         return True
-    if provider and provider.strip().lower() == "gateway" and not server:
+    if is_gateway and not server:
         return _is_local_gateway_configured(settings)
     return False
 
@@ -584,7 +593,9 @@ def calculate_request_cost(
         it and not at `model`, the route name that the client asked for, which genai-prices does
         not know. genai-prices takes the provider from the prefix of the reference (#2031).
         """
-        if is_local(server=server, provider=provider, served_by=served_by):
+        if is_local(
+            server=server, provider=provider, served_by=served_by, served_model=served_model
+        ):
             return RequestCost(0.0, CONST_PRICING_SOURCE_LOCAL)
         if served_model:
             model, provider = served_model, CONST_GENAI_PRICES_LITELLM_PROVIDER

@@ -301,7 +301,8 @@ class AIMessages:
     )
     hosted_model_unpriced: str = (
         "genai-prices and the price overrides hold no price for model {model} under provider "
-        "{provider}. Its requests are recorded at $0 with the pricing source unknown."
+        "{provider}. Its requests are recorded at $0 with the pricing source unknown. To set a "
+        "price, run 'devops ai cost set-price {model} <prompt_rate> <completion_rate>'."
     )
     reference_model_unpriced: str = (
         "genai-prices and the price overrides hold no price for {model} ({setting}) under "

@@ -2042,6 +2042,19 @@ def _hosted_violations(config: dict[str, Any], deployment: dict[str, Any]) -> li
     return problems
 
 
+def test_every_local_deployment_names_its_backend_so_the_ledger_counts_it_as_local() -> None:
+    """A gateway reply names a deployment model and no backend only for a deployment with no
+    `api_base`, and the spend ledger counts such a call as hosted (#3656). So each deployment of
+    the in-cluster groups sets `api_base`, and each hosted group sets none."""
+    config = _full_gateway_config()
+
+    no_backend = {
+        m["model_name"] for m in config["model_list"] if not m["litellm_params"].get("api_base")
+    }
+
+    assert no_backend == set(HOSTED_GROUPS)
+
+
 def test_the_hosted_groups_call_providers_with_optional_keys_and_join_no_pool() -> None:
     """The hosted groups follow their rules (#3113)."""
     deployment = _load_kind(GATEWAY_DIR / "deployment.yaml", "Deployment")

@@ -1036,6 +1036,31 @@ def test_the_report_counts_a_gateway_call_as_local_by_the_backend_that_served_it
     ) == (1, 1_000_000, 2.5, [("localhost:4000", False)])
 
 
+def test_the_report_counts_a_hosted_group_that_names_no_backend_as_hosted(tmp_path: Path) -> None:
+    """A hosted group sets no `api_base`, so its reply names a deployment model and no backend.
+    Of two such calls through one loopback gateway, the call of a model with a deployment model
+    is hosted and does not count in local savings, and the call that names none stays with the
+    host that decides."""
+    ledger = SpendLedger(db_path=tmp_path / "spend.db")
+    for model, served_model, cost in (
+        ("claude-haiku-5-5", "anthropic/claude-haiku-5-5", 0.6),
+        ("devops-chat", None, 0.0),
+    ):
+        ledger.record_request(
+            provider="gateway",
+            server="localhost:4000",
+            served_model=served_model,
+            model=model,
+            prompt_tokens=1_000_000,
+            completion_tokens=100_000,
+            cost_usd=cost,
+        )
+
+    report = ledger.get_lifetime_report(reference_model="gpt-4o")
+
+    assert (report.local_requests, report.local_prompt_tokens) == (1, 1_000_000)
+
+
 def _local_chat_and_embedding_ledger(db_path: Path) -> SpendLedger:
     """A ledger with one local chat request and one local embedding request, of 1M tokens each."""
     ledger = SpendLedger(db_path=db_path)