diff --git a/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/.openspec.yaml b/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/.openspec.yaml new file mode 100644 index 000000000..1491b921e --- /dev/null +++ b/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-11-05 diff --git a/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/proposal.md b/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/proposal.md new file mode 100644 index 000000000..99fe64830 --- /dev/null +++ b/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/proposal.md @@ -0,0 +1,31 @@ +# Proposal + +## Why + +Issue #3656: a call to a hosted gateway group (`claude-sonnet-5-5`, `claude-haiku-5-5`, `deepseek-v4.1-flash-1017`) is recorded as a local call at $0, and its tokens count as local savings. The ledger of the owner holds 354 such rows. The hosted groups set no `api_base`, so the gateway reply names no backend, and `is_local` then decides from the server that the call was sent to. That server is the gateway, a local host. The reply does name the deployment model (`x-litellm-model-name`), and `is_local` does not read it. + +A second gap: genai-prices holds no price for `deepseek-v4.1-flash-1017`. The warning for it does not say how to set a price. + +## Feasibility + +- The local ledger (`.data/ai/spend.db`, 2026-11-05) groups the rows with a deployment model that starts with `anthropic/` or `openrouter/`: 354 rows, `served_by` NULL in all, `pricing_source` `local` in all. Every row with a `served_by` is a cluster backend. The query is in issue #3656. +- `genai_prices.calc_price` (genai-prices 0.1.13) prices `claude-sonnet-5-5` and `claude-haiku-5-5` at the provider `anthropic` (2.0 and 10.0 USD per million tokens for Sonnet, the prices of the gateway group), and `gemini-3.8-flash` at the provider `google`. It raises `LookupError` for `deepseek-v4.1-flash-1017` under the providers `litellm` and `openrouter`. +- The upstream data (`https://raw.githubusercontent.com/pydantic/genai-prices/refs/heads/main/prices/new_data/v2/data.json`, 2026-11-05) lists `deepseek/deepseek-v4.1-flash` under `openrouter` and no `-1017` snapshot. Its price (0.0352 to 0.0448 USD input, 0.08 USD output per million) is not the price of the group (0.05 and 0.192 in `k8s/llm/gateway/config.yaml`), so a lookup by the base name would price the group wrongly. +- `devops ai gateway render` and `tests/ai/benchmark/test_k8s_llm_gateway.py` already treat a deployment with no `api_base` as hosted. + +## Existing Tool Considered + +genai-prices prices the Claude and Gemini groups once the ledger asks it. For the DeepSeek group the open mechanism is the price override that the repository already has (`devops ai cost set-price`), keyed by the deployment model. The change builds no price table and no model-name matching. It makes the warning name the command. It deletes no code. + +## What Changes + +- `is_local` takes `served_model`. A reply of the `gateway` provider that names a deployment model and no backend is hosted. `calculate_request_cost` passes the deployment model, and the lifetime report groups the served usage by it, so a hosted group no longer counts in local savings or sets `local` on its tokens. +- The warning for an unpriced hosted model adds the command `devops ai cost set-price `. +- `tests/ai/benchmark/test_k8s_llm_gateway.py` fails when an in-cluster deployment sets no `api_base`: the ledger would count its calls as hosted. +- The spec delta adds two requirements to `ai-spend-pricing`. +- The change fits one pull request. + +## Impact + +- Rows that the ledger recorded before this change keep the source `local`. No migration: a row records what the process knew when it wrote it. +- The DeepSeek group has the source `unknown` and a $0 cost until a person sets a price with the command that the warning names. The three known prices differ (the gateway config, the OpenRouter list, genai-prices), so the change chooses none. diff --git a/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/specs/ai-spend-pricing/spec.md b/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/specs/ai-spend-pricing/spec.md new file mode 100644 index 000000000..36002f030 --- /dev/null +++ b/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/specs/ai-spend-pricing/spec.md @@ -0,0 +1,27 @@ +## ADDED Requirements + +### Requirement: A gateway reply that names a deployment model and no backend is hosted +A reply of the `gateway` provider that names a deployment model in the `x-litellm-model-name` header and names no backend in the `x-litellm-model-api-base` header SHALL count as a hosted call, whatever host the gateway has. A reply that names a backend SHALL stay decided by that backend. The lifetime report SHALL apply the same rule to the rows of the ledger, and a row of a hosted call SHALL NOT count in local savings. + +#### Scenario: Hosted groups behind a loopback gateway +- **WHEN** replies to `claude-haiku-5-5`, `claude-sonnet-5-5` and `gemini-3.8-flash` through a gateway on `127.0.0.1` name `anthropic/claude-haiku-5-5`, `anthropic/claude-sonnet-5-5` and `gemini/gemini-3.8-flash` and no backend +- **THEN** the pricing sources are `genai_prices:anthropic`, `genai_prices:anthropic` and `genai_prices:google`, and each cost is more than $0 + +#### Scenario: A reply that names a local backend +- **WHEN** a reply names a deployment model and a cluster Ollama as its backend +- **THEN** the call is local + +#### Scenario: The report counts a hosted group as hosted +- **WHEN** the ledger holds one call that names `anthropic/claude-haiku-5-5` and no backend, and one call that names neither, both sent to `localhost:4000` through the `gateway` provider +- **THEN** the report counts one local request + +### Requirement: The warning for an unpriced hosted model names the command that sets a price +The warning that names a hosted model that neither genai-prices nor a price override prices SHALL give the command `devops ai cost set-price ` with the model of the warning. + +#### Scenario: The DeepSeek group has no price +- **WHEN** two replies name `openrouter/deepseek/deepseek-v4.1-flash-1017:floor` and no backend, and no price override exists +- **THEN** both rows have the source `unknown` and a cost of $0, and one warning names the command `devops ai cost set-price` + +#### Scenario: A price override of the deployment model +- **WHEN** a price override of `deepseek-v4.1-flash-1017:floor` is 0.05 USD input and 0.192 USD output per million tokens +- **THEN** a call of 1,000,000 prompt and 1,000,000 completion tokens costs 0.242 USD with the source `custom_override` diff --git a/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/tasks.md b/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/tasks.md new file mode 100644 index 000000000..b94b7204b --- /dev/null +++ b/openspec/changes/archive/2026-11-05-3656-hosted-gateway-priced-local/tasks.md @@ -0,0 +1,18 @@ +# Tasks + +## 1. Count a hosted gateway group as hosted + +Kind: behavior. + +Reads: `src/devops_cli/ai/spend/pricing.py` (`is_local`, `calculate_request_cost`), `src/devops_cli/ai/spend/ledger.py` (`_QUERY_SERVED_USAGE`, `_tally_and_annotate_servers`), `src/devops_cli/lang/en/messages.py` (`hosted_model_unpriced`). Changes: the same files, `tests/ai/spend/test_ai_spend_pricing.py`, `tests/ai/spend/test_ai_spend_ledger.py` and `tests/ai/benchmark/test_k8s_llm_gateway.py`. + +- [x] 1.1 Add the failing tests: `is_local` of a reply with a deployment model and no backend, the price of the Claude and Gemini groups behind a loopback gateway, the warning and the override of the DeepSeek group, the lifetime report, and the gateway config check that each local deployment sets `api_base` +- [x] 1.2 Make `is_local` read the deployment model, pass it from `calculate_request_cost` and the report, and name the command in the warning + - Verification: `uv run devops ci test tests/ai/spend tests/ai/benchmark/test_k8s_llm_gateway.py` +- [x] 1.3 Regenerate the documentation with `uv run devops docs generate --sync-readme`. No generated document holds the message, so no file changes + - Verification: `uv run devops ci docs` + +## Workflow follow-up + +- Pending a person: set the price of the group `deepseek-v4.1-flash-1017` with `devops ai cost set-price`. +- Pending a person: review the diff in tuicr (tuicr -r origin/main..HEAD). diff --git a/openspec/specs/ai-spend-pricing/spec.md b/openspec/specs/ai-spend-pricing/spec.md index b865e6a94..6d312e5ed 100644 --- a/openspec/specs/ai-spend-pricing/spec.md +++ b/openspec/specs/ai-spend-pricing/spec.md @@ -158,3 +158,29 @@ When a chat call that returns one reply fails after its status line, because of #### Scenario: An error frame after a ping - **WHEN** a chat call gets a stream with a `: ping` comment and then an error frame with `code` `"500"` - **THEN** the spend ledger has one row with the request type `chat_dispatch` and finish reason `error` + +### Requirement: A gateway reply that names a deployment model and no backend is hosted +A reply of the `gateway` provider that names a deployment model in the `x-litellm-model-name` header and names no backend in the `x-litellm-model-api-base` header SHALL count as a hosted call, whatever host the gateway has. A reply that names a backend SHALL stay decided by that backend. The lifetime report SHALL apply the same rule to the rows of the ledger, and a row of a hosted call SHALL NOT count in local savings. + +#### Scenario: Hosted groups behind a loopback gateway +- **WHEN** replies to `claude-haiku-5-5`, `claude-sonnet-5-5` and `gemini-3.8-flash` through a gateway on `127.0.0.1` name `anthropic/claude-haiku-5-5`, `anthropic/claude-sonnet-5-5` and `gemini/gemini-3.8-flash` and no backend +- **THEN** the pricing sources are `genai_prices:anthropic`, `genai_prices:anthropic` and `genai_prices:google`, and each cost is more than $0 + +#### Scenario: A reply that names a local backend +- **WHEN** a reply names a deployment model and a cluster Ollama as its backend +- **THEN** the call is local + +#### Scenario: The report counts a hosted group as hosted +- **WHEN** the ledger holds one call that names `anthropic/claude-haiku-5-5` and no backend, and one call that names neither, both sent to `localhost:4000` through the `gateway` provider +- **THEN** the report counts one local request + +### Requirement: The warning for an unpriced hosted model names the command that sets a price +The warning that names a hosted model that neither genai-prices nor a price override prices SHALL give the command `devops ai cost set-price ` with the model of the warning. + +#### Scenario: The DeepSeek group has no price +- **WHEN** two replies name `openrouter/deepseek/deepseek-v4.1-flash-1017:floor` and no backend, and no price override exists +- **THEN** both rows have the source `unknown` and a cost of $0, and one warning names the command `devops ai cost set-price` + +#### Scenario: A price override of the deployment model +- **WHEN** a price override of `deepseek-v4.1-flash-1017:floor` is 0.05 USD input and 0.192 USD output per million tokens +- **THEN** a call of 1,000,000 prompt and 1,000,000 completion tokens costs 0.242 USD with the source `custom_override` diff --git a/src/devops_cli/ai/spend/ledger.py b/src/devops_cli/ai/spend/ledger.py index 6989cb85a..b28f8b725 100644 --- a/src/devops_cli/ai/spend/ledger.py +++ b/src/devops_cli/ai/spend/ledger.py @@ -146,19 +146,21 @@ """ # The backend that served a call decides whether it ran locally, so local usage is tallied by -# server, provider and serving backend (#1631). +# server, provider and serving backend (#1631). A gateway reply that names a deployment model and +# no backend is hosted, so the tally also keeps the deployment model (#3656). _QUERY_SERVED_USAGE = """ SELECT server, provider, served_by, + served_model, COUNT(*) as req_count, COALESCE(SUM(prompt_tokens), 0) as p_tokens, COALESCE(SUM(completion_tokens), 0) as c_tokens, COALESCE(SUM(CASE WHEN request_type = :embedding THEN prompt_tokens ELSE 0 END), 0) as e_tokens FROM ai_spend_records WHERE (:cutoff IS NULL OR timestamp >= :cutoff) -GROUP BY server, provider, served_by; +GROUP BY server, provider, served_by, served_model; """ _QUERY_MODEL_BREAKDOWN = """ @@ -319,6 +321,7 @@ class _ServedUsage(NamedTuple): server: str provider: str served_by: str | None + served_model: str | None usage: _Usage @@ -339,7 +342,10 @@ def _tally_and_annotate_servers( hosted: set[tuple[str, str]] = set() for group in served: if group.server != CONST_RESPONSE_CACHE_SERVER and is_local( - server=group.server, provider=group.provider, served_by=group.served_by + server=group.server, + provider=group.provider, + served_by=group.served_by, + served_model=group.served_model, ): local.append(group.usage) else: @@ -663,6 +669,7 @@ def _query_served_usage( server=r["server"], provider=r["provider"], served_by=r["served_by"], + served_model=r["served_model"], usage=_Usage( requests=int(r["req_count"] or 0), prompt_tokens=int(r["p_tokens"] or 0), diff --git a/src/devops_cli/ai/spend/pricing.py b/src/devops_cli/ai/spend/pricing.py index f07ea3e4d..da3925a62 100644 --- a/src/devops_cli/ai/spend/pricing.py +++ b/src/devops_cli/ai/spend/pricing.py @@ -15,6 +15,7 @@ from devops_cli.ai.spend.models import ModelPricing from devops_cli.config.constants import ( + CONST_AI_GATEWAY_PROVIDER, CONST_GENAI_PRICES_LITELLM_PROVIDER, CONST_LOCAL_PROVIDER_NAMES, CONST_LOCAL_TRANSPORT_LABELS, @@ -274,26 +275,34 @@ def _is_local_gateway_configured(settings: _SettingsOnce) -> bool: def is_local( - server: str | None = None, provider: str | None = None, served_by: str | None = None + server: str | None = None, + provider: str | None = None, + served_by: str | None = None, + served_model: str | None = None, ) -> bool: """Return True if the call ran on local hardware. `served_by` is the backend that served the call, as the LLM gateway names it in its reply (`ai/client/network.py` reads the header). When the reply names it, it decides: a hosted route behind a local gateway is hosted, and a local backend behind a gateway on a public host is - local (#1631). LiteLLM names the API base of a route only when it knows it, so a gateway - route sets `api_base`. When the reply names no backend, the provider and the server the call - was sent to decide. The call loads the settings at most once, and only for a host that its - address and its name do not decide (#1420). + local (#1631). LiteLLM names the API base of a route only when it knows it, so a local route + sets `api_base` and a hosted group sets none. So a gateway reply that names a deployment + model, `served_model`, and no backend is hosted, whatever host the gateway has (#3656). When + the reply names neither, the provider and the server the call was sent to decide. The call + loads the settings at most once, and only for a host that its address and its name do not + decide (#1420). """ settings = _SettingsOnce() if served_by and served_by.strip(): return _is_local_server(served_by, settings) + is_gateway = provider is not None and provider.strip().lower() == CONST_AI_GATEWAY_PROVIDER + if is_gateway and served_model and served_model.strip(): + return False if _is_local_provider(provider): return True if _is_local_server(server, settings): return True - if provider and provider.strip().lower() == "gateway" and not server: + if is_gateway and not server: return _is_local_gateway_configured(settings) return False @@ -584,7 +593,9 @@ def calculate_request_cost( it and not at `model`, the route name that the client asked for, which genai-prices does not know. genai-prices takes the provider from the prefix of the reference (#2031). """ - if is_local(server=server, provider=provider, served_by=served_by): + if is_local( + server=server, provider=provider, served_by=served_by, served_model=served_model + ): return RequestCost(0.0, CONST_PRICING_SOURCE_LOCAL) if served_model: model, provider = served_model, CONST_GENAI_PRICES_LITELLM_PROVIDER diff --git a/src/devops_cli/lang/en/messages.py b/src/devops_cli/lang/en/messages.py index 645ba6aa7..64f3a3b34 100644 --- a/src/devops_cli/lang/en/messages.py +++ b/src/devops_cli/lang/en/messages.py @@ -301,7 +301,8 @@ class AIMessages: ) hosted_model_unpriced: str = ( "genai-prices and the price overrides hold no price for model {model} under provider " - "{provider}. Its requests are recorded at $0 with the pricing source unknown." + "{provider}. Its requests are recorded at $0 with the pricing source unknown. To set a " + "price, run 'devops ai cost set-price {model} '." ) reference_model_unpriced: str = ( "genai-prices and the price overrides hold no price for {model} ({setting}) under " diff --git a/tests/ai/benchmark/test_k8s_llm_gateway.py b/tests/ai/benchmark/test_k8s_llm_gateway.py index f57b72141..e3db44e2a 100644 --- a/tests/ai/benchmark/test_k8s_llm_gateway.py +++ b/tests/ai/benchmark/test_k8s_llm_gateway.py @@ -2042,6 +2042,19 @@ def _hosted_violations(config: dict[str, Any], deployment: dict[str, Any]) -> li return problems +def test_every_local_deployment_names_its_backend_so_the_ledger_counts_it_as_local() -> None: + """A gateway reply names a deployment model and no backend only for a deployment with no + `api_base`, and the spend ledger counts such a call as hosted (#3656). So each deployment of + the in-cluster groups sets `api_base`, and each hosted group sets none.""" + config = _full_gateway_config() + + no_backend = { + m["model_name"] for m in config["model_list"] if not m["litellm_params"].get("api_base") + } + + assert no_backend == set(HOSTED_GROUPS) + + def test_the_hosted_groups_call_providers_with_optional_keys_and_join_no_pool() -> None: """The hosted groups follow their rules (#3113).""" deployment = _load_kind(GATEWAY_DIR / "deployment.yaml", "Deployment") diff --git a/tests/ai/spend/test_ai_spend_ledger.py b/tests/ai/spend/test_ai_spend_ledger.py index 475a0ecbc..f8adf3bd8 100644 --- a/tests/ai/spend/test_ai_spend_ledger.py +++ b/tests/ai/spend/test_ai_spend_ledger.py @@ -1036,6 +1036,31 @@ def test_the_report_counts_a_gateway_call_as_local_by_the_backend_that_served_it ) == (1, 1_000_000, 2.5, [("localhost:4000", False)]) +def test_the_report_counts_a_hosted_group_that_names_no_backend_as_hosted(tmp_path: Path) -> None: + """A hosted group sets no `api_base`, so its reply names a deployment model and no backend. + Of two such calls through one loopback gateway, the call of a model with a deployment model + is hosted and does not count in local savings, and the call that names none stays with the + host that decides.""" + ledger = SpendLedger(db_path=tmp_path / "spend.db") + for model, served_model, cost in ( + ("claude-haiku-5-5", "anthropic/claude-haiku-5-5", 0.6), + ("devops-chat", None, 0.0), + ): + ledger.record_request( + provider="gateway", + server="localhost:4000", + served_model=served_model, + model=model, + prompt_tokens=1_000_000, + completion_tokens=100_000, + cost_usd=cost, + ) + + report = ledger.get_lifetime_report(reference_model="gpt-4o") + + assert (report.local_requests, report.local_prompt_tokens) == (1, 1_000_000) + + def _local_chat_and_embedding_ledger(db_path: Path) -> SpendLedger: """A ledger with one local chat request and one local embedding request, of 1M tokens each.""" ledger = SpendLedger(db_path=db_path) diff --git a/tests/ai/spend/test_ai_spend_pricing.py b/tests/ai/spend/test_ai_spend_pricing.py index 4207d8b5a..e186771fe 100644 --- a/tests/ai/spend/test_ai_spend_pricing.py +++ b/tests/ai/spend/test_ai_spend_pricing.py @@ -721,6 +721,86 @@ def test_a_gateway_call_is_local_by_the_backend_that_served_it( ] == [True, True, False, True, False] +def test_a_gateway_reply_that_names_a_deployment_model_and_no_backend_is_hosted( + config: Callable[..., Path], +) -> None: + """Verify a hosted group sets no `api_base`, so its reply names a deployment model and no + backend. That call is hosted behind a loopback gateway, a configured gateway host and a + gateway with no server, while a reply that names a local backend stays local.""" + from devops_cli.ai.spend.pricing import is_local + + config(k8s={"domain": "example.com"}, ai={"gateway_url": "https://gw.example.com/v1"}) + claude = "anthropic/claude-haiku-5-5" + cluster_ollama = "http://ollama-16gib.llm.svc.cluster.local:11434" + + assert [ + is_local("localhost:4000", "gateway", served_model=claude), + is_local("gw.example.com", "gateway", served_model=claude), + is_local(None, "gateway", served_model=claude), + is_local("localhost:4000", "gateway", served_by=cluster_ollama, served_model=claude), + is_local("localhost:4000", "gateway", served_model=" "), + ] == [False, False, False, True, True] + + +@pytest.mark.parametrize( + ("model", "served_model", "source"), + [ + ("claude-haiku-5-5", "anthropic/claude-haiku-5-5", "genai_prices:anthropic"), + ("claude-sonnet-5-5", "anthropic/claude-sonnet-5-5", "genai_prices:anthropic"), + ("gemini-3.8-flash", "gemini/gemini-3.8-flash", "genai_prices:google"), + ], +) +def test_a_hosted_group_behind_a_loopback_gateway_is_priced_by_genai_prices( + tmp_path: Path, model: str, served_model: str, source: str +) -> None: + """Verify the Claude and Gemini groups of the gateway cost money: the reply names the + deployment model and no backend, and genai-prices prices the model.""" + registry = PricingRegistry(data_dir=tmp_path) + + cost = registry.calculate_request_cost( + model, + 1_000_000, + 100_000, + server="127.0.0.1:4200", + provider="gateway", + served_model=served_model, + ) + + assert (cost.pricing_source, cost.cost_usd > 0) == (source, True) + + +def test_a_hosted_group_that_nothing_prices_warns_with_the_command_that_sets_a_price( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """Verify the DeepSeek group, which genai-prices does not list, is hosted at $0 with the + source `unknown`, and that its one warning names the command that sets a price, and a price + override of the deployment model prices it.""" + deepseek = "openrouter/deepseek/deepseek-v4.1-flash-1017:floor" + registry = PricingRegistry(data_dir=tmp_path) + call = { + "model": "deepseek-v4.1-flash-1017", + "server": "127.0.0.1:4200", + "provider": "gateway", + "served_model": deepseek, + } + + with caplog.at_level(logging.WARNING, logger="devops_cli.ai.spend.pricing"): + unpriced = [registry.calculate_request_cost(**call, prompt_tokens=10, completion_tokens=1)] + unpriced.append( + registry.calculate_request_cost(**call, prompt_tokens=10, completion_tokens=1) + ) + registry.set_custom_pricing("deepseek-v4.1-flash-1017:floor", 0.05, 0.192) + priced = registry.calculate_request_cost( + **call, prompt_tokens=1_000_000, completion_tokens=1_000_000 + ) + + assert ( + [(c.cost_usd, c.pricing_source) for c in unpriced], + [r.getMessage().count("devops ai cost set-price") for r in caplog.records], + (priced.cost_usd, priced.pricing_source), + ) == ([(0.0, "unknown")] * 2, [1], (0.242, "custom_override")) + + def _settings_loads_of(call: Callable[[], object]) -> int: """The number of times `call` runs `load_settings`, counted by a profile hook, so the test patches no devops-cli name (#1420).""" @@ -885,7 +965,7 @@ def price(model: str, server: str, provider: str) -> RequestCost: RequestCost(2.5, "genai_prices:openai"), RequestCost(0.0, "local"), [ - "genai-prices and the price overrides hold no price for model gpt-9-preview under provider openai. Its requests are recorded at $0 with the pricing source unknown." + "genai-prices and the price overrides hold no price for model gpt-9-preview under provider openai. Its requests are recorded at $0 with the pricing source unknown. To set a price, run 'devops ai cost set-price gpt-9-preview '." ], ) @@ -942,7 +1022,7 @@ def price(served_model: str | None, served_by: str = "https://api.openai.com") - ], RequestCost(0.0, "local"), [ - "genai-prices and the price overrides hold no price for model openai/gpt-9-preview under provider litellm. Its requests are recorded at $0 with the pricing source unknown." + "genai-prices and the price overrides hold no price for model openai/gpt-9-preview under provider litellm. Its requests are recorded at $0 with the pricing source unknown. To set a price, run 'devops ai cost set-price openai/gpt-9-preview '." ], )