Skip to content

feat(gateway): per-model context limits - #337

Merged
quickbeard merged 5 commits into
mainfrom
gateway-model-limits
Aug 27, 2026
Merged

quickbeard merged 5 commits into
mainfrom
gateway-model-limits

Conversation

@quickbeard

@quickbeard quickbeard commented Aug 27, 2026 •

Copy link
Copy Markdown
Owner

What

Ports the hub's per-model limit scheme into packages/codev-gateway, replacing the flat GATEWAY_CONTEXT_WINDOW = 196608 that stamped every gateway model with the same window.

The flat constant claimed to be "identical to the hub's constants so a hub-written and an agent-written codev.json agree byte-for-byte" — no longer true since the hub moved to per-model limits (quickbeard/codev-hub#259 keeps the two in lockstep). A config written by the TUI /configure dialog, codev configure, or the desktop gateway sign-in gave every model the same 196,608-token window and the same compaction point.

How

  • model-limits.ts (new): ModelLimits { context, trigger, output? } resolved remote → table → default like the hub — gateway-reported windows first, then the static window table (MiniMax/MiniMax-M3 at 262,144 — the deployment's resource cap, not the model's native window — and zai-org/GLM-4.7-cc at 200K), then the safe 200K default for unknown ids. The trigger is uniformly 90% of the window, whatever its source. declaredInput() implements the hub's limit.input = trigger + reserved trick (clamped to context), which lands each model's own compaction trigger against the single global compaction.reserved: 40000 — opencode's schema has no per-model compaction knob (session/overflow.ts: usable = limit.input − reserved).
  • config-block.ts: buildGatewayConfig / buildCustomProviderConfig now write per-model limit.context/input/output via limitsFor (M3 gets context: 262144; its 90% target + reserve exceeds the window, so the clamp pins input: 262144 — effective trigger 222,144); compaction.reserved becomes 40000, matching the hub's writer.
  • backend.ts: fetchModelWindows() reads LiteLLM's /model_group/info at the gateway root — never throws, returns {} when the gateway reports nothing (today's deployment: every max_input_tokens is null, so the static table carries all models; the moment an admin fills the field in, that model becomes gateway-driven with no release).
  • auth-file.ts: model_limits cache in the shared auth file — the same key the hub's saveModelLimits writes, so hub-cached windows are honored here and vice versa.
  • Wiring: the TUI configure dialog and the desktop gateway-sso controller cache windows right after their existing fetchModels step, non-fatally.

The custom-provider block deliberately still writes no compaction key (connecting a provider must not change global compaction for unrelated ones); declaredInput's clamp keeps its declared budget inside the true window.

All touched files are fork-only, so no DIVERGENCES.md rows.

Tests

  • packages/codev-gateway: 52 pass (new model-limits.test.ts; fetchModelWindows endpoint/skip/never-throw cases; config-block expectations updated + hermetic CODEV_TEST_HOME so a hub-cached model_limits on a dev machine can't leak in).
  • packages/desktop gateway-sso controller: 14 pass (asserts windows are cached during the models step).
  • packages/tui custom-provider view test: 6 pass.
  • bun typecheck clean in codev-gateway, tui, desktop, app, opencode.

🤖 Generated with Claude Code

Quickbeard and others added 4 commits August 27, 2026 10:05
Port the hub's per-model limit scheme (hub src/lib/model-limits.ts) into
packages/codev-gateway, replacing the flat 196608-token window that assumed
every gateway model was the same size — untrue once the gateway served both
a 1M-token and a 200K-token model.

- model-limits.ts: ModelLimits {context, trigger, output?}, resolved
  remote -> table -> 200K default; declaredInput() lands each model's own
  compaction trigger against the single global compaction.reserved (40000).
- Config blocks now write per-model limit.context/input/output; the 1M model
  gets its true window so the context gauge and compaction stop understating
  it by ~5x.
- fetchModelWindows() reads LiteLLM's /model_group/info (never throws);
  the TUI and desktop configure flows cache the result in the shared auth
  file under model_limits — the same key the hub writes — so hub-cached
  windows are honored and vice versa.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The table now carries only context windows — the trigger is always 90% of
the window regardless of source (table, default, or gateway-reported) —
and the 1M model's id is a plain string instead of base64.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gateway serves MiniMax-M3 with a 262144-token window (resource cap),
not the model's native 1M. With the 90% trigger + 40000 reserve exceeding
that window, declaredInput's clamp pins limit.input at the window itself.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@quickbeard
quickbeard merged commit b67d8a3 into main Aug 27, 2026
0 of 5 checks passed
@quickbeard
quickbeard deleted the gateway-model-limits branch August 27, 2026 04:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant