Repository navigation
feat(gateway): per-model context limits - #337
Merged
Merged
Conversation
Port the hub's per-model limit scheme (hub src/lib/model-limits.ts) into
packages/codev-gateway, replacing the flat 196608-token window that assumed
every gateway model was the same size — untrue once the gateway served both
a 1M-token and a 200K-token model.
- model-limits.ts: ModelLimits {context, trigger, output?}, resolved
remote -> table -> 200K default; declaredInput() lands each model's own
compaction trigger against the single global compaction.reserved (40000).
- Config blocks now write per-model limit.context/input/output; the 1M model
gets its true window so the context gauge and compaction stop understating
it by ~5x.
- fetchModelWindows() reads LiteLLM's /model_group/info (never throws);
the TUI and desktop configure flows cache the result in the shared auth
file under model_limits — the same key the hub writes — so hub-cached
windows are honored and vice versa.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The table now carries only context windows — the trigger is always 90% of the window regardless of source (table, default, or gateway-reported) — and the 1M model's id is a plain string instead of base64. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gateway serves MiniMax-M3 with a 262144-token window (resource cap), not the model's native 1M. With the 90% trigger + 40000 reserve exceeding that window, declaredInput's clamp pins limit.input at the window itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports the hub's per-model limit scheme into
packages/codev-gateway, replacing the flatGATEWAY_CONTEXT_WINDOW = 196608that stamped every gateway model with the same window.The flat constant claimed to be "identical to the hub's constants so a hub-written and an agent-written codev.json agree byte-for-byte" — no longer true since the hub moved to per-model limits (quickbeard/codev-hub#259 keeps the two in lockstep). A config written by the TUI
/configuredialog,codev configure, or the desktop gateway sign-in gave every model the same 196,608-token window and the same compaction point.How
model-limits.ts(new):ModelLimits { context, trigger, output? }resolved remote → table → default like the hub — gateway-reported windows first, then the static window table (MiniMax/MiniMax-M3 at 262,144 — the deployment's resource cap, not the model's native window — and zai-org/GLM-4.7-cc at 200K), then the safe 200K default for unknown ids. The trigger is uniformly 90% of the window, whatever its source.declaredInput()implements the hub'slimit.input = trigger + reservedtrick (clamped tocontext), which lands each model's own compaction trigger against the single globalcompaction.reserved: 40000— opencode's schema has no per-model compaction knob (session/overflow.ts:usable = limit.input − reserved).config-block.ts:buildGatewayConfig/buildCustomProviderConfignow write per-modellimit.context/input/outputvialimitsFor(M3 getscontext: 262144; its 90% target + reserve exceeds the window, so the clamp pinsinput: 262144— effective trigger 222,144);compaction.reservedbecomes 40000, matching the hub's writer.backend.ts:fetchModelWindows()reads LiteLLM's/model_group/infoat the gateway root — never throws, returns{}when the gateway reports nothing (today's deployment: everymax_input_tokensis null, so the static table carries all models; the moment an admin fills the field in, that model becomes gateway-driven with no release).auth-file.ts:model_limitscache in the shared auth file — the same key the hub'ssaveModelLimitswrites, so hub-cached windows are honored here and vice versa.fetchModelsstep, non-fatally.The custom-provider block deliberately still writes no
compactionkey (connecting a provider must not change global compaction for unrelated ones);declaredInput's clamp keeps its declared budget inside the true window.All touched files are fork-only, so no DIVERGENCES.md rows.
Tests
packages/codev-gateway: 52 pass (newmodel-limits.test.ts;fetchModelWindowsendpoint/skip/never-throw cases; config-block expectations updated + hermeticCODEV_TEST_HOMEso a hub-cachedmodel_limitson a dev machine can't leak in).packages/desktopgateway-sso controller: 14 pass (asserts windows are cached during the models step).packages/tuicustom-provider view test: 6 pass.bun typecheckclean in codev-gateway, tui, desktop, app, opencode.🤖 Generated with Claude Code