Repository navigation
fix(server): fail over model limits and avoid cooling accounts - #88
Conversation
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
chrisohearn
left a comment
There was a problem hiding this comment.
Independent Claude Opus review of 11694dc (review of record)
Read-only review of the diff against 7836910, checked against real limit events in the provider logs and statev2. No tests run.
P1. Item 1 never fires on the real model-limit failures it cites. apps/server/src/provider/claudeUsageLimits.ts:33-35. All four cited failures (11-12 05:49, 06:09, 20:44, 23:18Z) arrive from Claude as api_error: "model_requires_usage_credits" with a rejected rateLimitType: "seven_day_overage_included". In each, five_hour is allowed and seven_day is only at allowed_warning. The result text is "You're out of usage credits. Switch to another model…". The adapter adds that raw type to rejectedRateLimitTypes (ClaudeAdapterV2.ts:9913-9916). The regex ^seven_day_(?!overage|extra) leaves out exactly that bucket. The prose regex matches only "7-day Fable/model limit", which is T3's own notice (ClaudeAdapterV2.ts:4701,4714), not Claude's text. So limitScope stays undefined, and acceptance query A can never show model. The tests use seven_day_fable, which is not in the SDK's rateLimitType union, plus T3's notice prose. Fix: classify seven_day_overage_included and/or api_error === "model_requires_usage_credits" as model, and test with the logged frame. The failure is safe, though: no account is cooled and no same-instance failover happens.
P2
- Lifted limits can't be cleared.
ProviderCooldown.ts:59-76has no manual clear and no upper limit on the reset time. An explicit pick of a cooled instance is overridden, and refreshing provider status doesn't clear it. Real case: the 11-13 04:22Z weekly limit (seven_day, resets 11-18) was lifted shortly afterwards. With this PR, every Claude prompt would have gone to the fallback for 5 days, until a restart or a successful wake on Claude. Suggest clearing on refresh or on an explicit pick, and capping at about 8 days. - The queued-start reroute has no failover/continuation guard.
Orchestrator.ts:2311-2320. The dispatch path has this guard (:7107), but queued runs can carryfailoverOfRunId(:7629,:7659). If its target cools before it starts, a failover run can be moved elsewhere, including back to the failing instance when that instance didn't cool (model scope, sharing cap). Restart continuations can be moved the same way. - A cooling primary hides the secondary.
ProviderFailover.ts:117with:81.available()turns a cooling or disabled same-instance primary intonullbefore the shares-instance check, so the secondary is skipped. Compare the raw primary's instance instead.
P3
- The reroute moves the thread to the fallback (
thread.provider-switched,Orchestrator.ts:2970-2980), and the thread stays there after the cooldown expires. That matches failover, but query C should allow for it. - The secondary is server-level only (
ProviderFailover.ts:118), while the primary resolves per project. - The PR body's item 4 quotes the old notice. The head prefixes
Codex instance "<id>":(CodexAdapterV2.ts:1197) and names namespace /additional_tools(CodexManagedErrors.ts:125,131).
Checks
(a) Holds for failover. The first-attempt, failover, failoverOfRunId, restartContinuationOfRunId and resendable checks are kept (ProviderFailover.ts:66-74). The claim still needs noWorkObserved and accountFailure. The outbox id effect:provider-failover:<run> is unchanged. The queued-reroute gap is listed under P2.
(b) Real account texts classify correctly: five_hour → "5-hour limit", seven_day → "weekly limit". Codex sets a reset only when a window is ≥100% used, so sharing caps get no reset and don't cool. Model texts are not classified (P1).
(c) The cooldown is bounded by its reset and cleared by success. getAll filters on the wall clock, so clock skew only shifts expiry toward the stated reset. With no fallback available, the run uses the cooled instance. PubSub: one layer-scoped subscription, and each stream consumer gets its own fromPubSub. No leak found. There is no Layer.fresh, so one cooldown instance is shared.
(d) No loop. A failover run can't fail over again, and cooling targets are excluded.
(e) Bounded. Retries cover only thread/resume on -32600 "active writer": 250/500/1000 ms, 4 attempts in all, all before turn/start.
(f) Older settings load: withDecodingDefault(null), plus an atomic patch key.
Verdict: blocking
… into r4/failover-provider-errors # Conflicts: # apps/server/src/orchestration-v2/Orchestrator.ts
11694dc to
87f3ff6
Compare
… into r4/failover-provider-errors # Conflicts: # apps/server/src/orchestration-v2/Orchestrator.ts # apps/server/src/provider/ProviderRegistry.ts # packages/contracts/src/orchestrationV2.ts
chrisohearn
left a comment
There was a problem hiding this comment.
Independent review of be7436f (review of record; follows 11694dc review, includes merge-resolution check of dc3ee55)
Read-only Claude Opus review run by the round-four coordinator. No tests run; hosted CI passed 19/19 on this head.
Checked
- History.
range-diff:e7a384fa37,8a9b000c26and83aaa49ee0are patch-identical to the 11694dc review.aabe025d8cis the only new non-merge commit. - Merges. Read both remerge-diffs:
87f3ff6ecc(thread freeze) andbe7436f464(dc3ee55). The Orchestrator queued start,thread.model-selection.set, ProviderRegistry (withCooldownsandobserveProviderbefore upsert) and the contracts (two optional payload fields, no migration) are all resolved correctly. - Earlier findings, now fixed:
- the Claude model-scoped limits;
- the cooldown clear: a 6 h cap, cleared on refresh, and cleared by a fresh usable probe only;
- the queued reroute guards;
- the cooling primary hiding the secondary.
- Loops. No retry or cooldown loops. A failover run never fails over again, and expiry is scoped.
P1. The merge discards the dispatch-time cooldown reroute. Orchestrator.ts:7473-7486 resolves the fallback. Then #97's :7805-7808 (auto-merged) restores nextTurnModelSelection(projection.thread) when command.modelSelection is undefined. That is exactly the non-bypass case. Scenario: instance X is cooling, and an MCP agent or a scheduled task sends with no model and start_immediately. The run is built on X, fails once, and only then fails over. No work is lost, but item 3 is void on this path. CI cannot catch it: no test sends a default start_immediately prompt during a cooldown. Fix: resolve the cooldown after :7807, on the selection the turn will actually use, and add that test.
P2. A scheduled pick gets no bypass. Orchestrator.ts:5264-5269: applyAt: next_turn on a busy thread records scheduledModelSelection without setting bypassProviderCooldown. So an explicit scheduled pick of a cooling instance is rerouted, and then the schedule is cleared. This is coupled with the P1: fixing the P1 alone overrides the pick on every path, so carry the bypass here too.
P2. Web, desktop and mobile sends always bypass the cooldown. Orchestrator.ts:7471-7472 sets the bypass whenever command.modelSelection is defined, and the composer always sends one (useComposerSend.ts:1305). Cooldown routing therefore only applies to sends that name no model. Fix: bypass only when the sent selection differs from the thread's default, or have the client mark a real pick. Otherwise correct the PR body's claim.
P3
- The failover rule (
ProviderFailover.ts:83: primary on the failing instance) differs from the cooldown-routing rule (:156: primary unusable). Confirm this is intended. defer_startis excluded from cooldown routing at:7476. Check that those runs are handled later.
Verdict: block: the merge with dc3ee55 (via #97's nextTurnModelSelection) overwrites the dispatch-time cooldown reroute, so default sends still go to cooling accounts first. Move the cooldown check after the scheduled-selection step and add a default start_immediately cooldown test.
… into r4/failover-provider-errors
chrisohearn
left a comment
There was a problem hiding this comment.
Independent delta review of 764da76 (review of record, follows be7436f review)
Read-only Claude Opus review run by the round-four coordinator. Hosted CI passed 19/19 on this head.
- History. One new non-merge commit,
f5607fe393(Orchestrator.ts plus the ProviderSwitch tests). The integration merge43ebf7443ehas an empty remerge-diff and touches only guard files. - Selection order. Cooldown routing now runs after the final next-turn and queue-tail selection (
:7776-7843), and nothing reassigns the selection after:7835.defer_startis now routed. Failover runs and restart continuations stay excluded. - Scheduled pick. The bypass is derived from the stored
scheduledModelSelection. A schedule always means a real, different pick, because:4612-4620stores none when the pick equals the current selection. - Deferred release. It starts the stored selection, so nothing is stranded.
resolveProviderCooldownTargetalways returns a selection, so there are no loops. - Tests exercise each claimed path: immediate default, scheduled, explicit, same and different composer selections, and deferred.
Earlier findings
- P1, default sends overwritten: resolved.
- P2, scheduled-pick bypass: resolved.
- P2, composer bypass: partial (see the P2 below).
- P3, rule difference: resolved as intended.
- P3,
defer_start: resolved.
P2 (follow-up): modelSelectionsEqual at :7822 is strict. The web composer adds fastMode: false while MCP and mobile selections omit it. So a web send of an unchanged model can still bypass once per thread per shape change. Fix: compare with draftModelPickMatchesThread, or normalize on the server.
P3s (follow-up):
- The
bypassProviderCooldownthread flag is cleared by automatic runs. - A lingering schedule can bypass repeatedly while held runs remain.
- The dispatch-side scheduled derivation and the multi-continuation claims have no direct tests.
Verdict: merge
f2acd00
into
integration/mcp-events-20261109
Current candidate verification
Head
764da7625ecd7db41f979dba69a977a416920658repairs the merge-resolution findings from the review ofbe7436f464e23f32643bfb153d675594b7022989and includes integration43ebf7443e13db63dbab04ada3a034467b5b4ce6. Pushed without force and remote-verified. The integration merge changed only host-guard files, not the checked runtime source. Hosted CI and a fresh independent review are the remaining gates.Current focused checks: 19 tests pass, comprising 12 provider-switch/routing/failover cases, 2 scheduled-model restoration cases, and 5 failover/cooldown unit cases. Five assertions failed before the repair: default immediate send, same-default composer send, same-default queued send, scheduled next-turn pick, and deferred preparation. All now pass. Scoped formatting and lint pass on the two changed files. Every heavy check used the shared resource lock and Vitest used
--maxWorkers=2 --no-file-parallelism. Hosted CI is the typecheck gate, as directed by the coordinator; local dependencies are stale after the upstream merge, so no dependency refresh or unrelated source repair was made.Earlier verification on
87f3ff6ecc292beec88b108112bf55e5803be945: before integration, 28 focused tests passed: 14 classifier/failover/cooldown cases, 4 real-native Claude scope cases, 7 failover/queued-routing integration cases, and 3 provider-registry race cases. Deleting the queued guards produced the 3 intended routing failures; restoring them passed. Scoped format/lint on all 16 repair files and server-only typecheck passed. After that integration, all 11 affected failover/queued-routing/freeze cases, scoped Orchestrator format/lint, and server-only typecheck passed.The earlier head
11694dc732a366a58394c5f862aae5d53ea578d1passed all 19 targeted Codex adapter classification scenarios and all 6 managed-error classifier tests, plus scoped format/lint. Its item-4 classification and notices are unchanged by the review repair.The preceding head
ec08bde9e09b85e16be63e7573c8608c74f724dfincludes the provider-status subscription-race repair and integration7836910e1c. All 76 tests in the three focused provider registry/compatibility/cooldown files passed there, including cooldown warning publication and clearing in the existing compatibility stream regression. Hosted CI is the full gate; the older check:changed results below apply only to their named earlier candidate. No broad local checks will be repeated under the resource rule.Problem
In the measured 48-hour window (1,473 runs), usage limits caused 19 of 38 failures but automatic failover fired once. The configured Claude fallback was excluded whenever it shared the failing instance, including when only the failed model was limited.
Related: #68 and #74; decision 0016. #74 remains open after this PR.
Items 1–5
1. Model-scoped limits
api_error=model_requires_usage_creditsand rejectedrateLimitType=seven_day_overage_included, while five-hour and seven-day account windows remain allowed. The native text says "You're out of usage credits. Switch to another model…".limitScope=model; query B must show a different target model (same instance is allowed). Historical rows are not rewritten.2. Secondary fallback
Evidence: 11 Claude five-hour account-limit failures had no eligible target because the primary fallback shared the failing instance.
Change: add
secondaryFallbackModelSelection, defaultnull. It is server-wide, not a per-project setting. It uses the same{ instanceId, model, options? }shape asfallbackModelSelection. Use it when the primary shares the failing instance and is ineligible, including a disabled or cooling primary; do not route to a disabled or cooling target. Preserve the configured primary's identity before checking availability. Settings replacement/clear semantics match the primary selection.Failover versus cooldown policy: the difference is intentional. After a failure,
providerFailoverTargettries the secondary only when the raw configured primary shares the failing instance, matching item 2. A disabled or cooling primary on a different instance does not trigger secondary failover. Before a new attempt,resolveProviderCooldownTargetinstead chooses the first available cross-instance target from primary then secondary, including when the primary is unset, disabled, cooling or on the source instance. This broader preflight rule avoids starting another prompt on an account already known to be cooling. Both paths exclude disabled/cooling targets and retain their respective resendability and run guards.Tests: target selection covers primary/secondary precedence and exclusions; the existing settings replacement/clear case now runs for both keys. The secondary replacement regression failed before the fix.
Acceptance: the coordinator configures the secondary (this PR does not change live settings), then reads it back with:
After a natural account-level failure, query B must name the secondary instance rather than another model on the limited instance.
3. Instance cooldown
Evidence: bursts of six to eight Claude account-limit failures within minutes on 2026-11-12 02:25–02:57 UTC.
Change: account-scoped usage failures with a valid future reset cool that instance until the earlier of the reset or six hours. Newly submitted and queued resendable prompts select an enabled, non-cooling fallback before creating another attempt. Failover runs, restart continuations, and already-claimed runs are not rerouted. Provider status shows the cooldown expiry and fallback notice.
Final selection: dispatch resolves cooldown only after applying the next-turn scheduled selection and queued-tail selection. A default
start_immediatelysend therefore starts directly on the fallback without first attempting the cooling source.Composer bypass rule: merely sending
modelSelectionis not an explicit override, because clients send it on ordinary prompts. A sent selection bypasses only when it differs from the thread's current default. An actual persisted model-picker action also bypasses. The existing persistedscheduledModelSelectionrecords an explicit next-turn pick; matching dispatches and queued starts honor it for as long as that schedule applies, without another flag or schema change.Deferred starts: standalone
defer_startruns becomepreparing, notqueued, andprepared-run.releasestarts their stored selection directly. They now receive cooldown routing at final dispatch, before preparation. A deferred send that queues behind an active run also gets the queued-start recheck. A limit learned only after standalone preparation began does not retroactively move that already-preparing run.Recovery: provider-status Refresh manually clears the selected instance's cooldown. A fresh authenticated ready/warning probe with usable account quota clears it automatically; stale, absent, unavailable or partial Claude quota does not. Claude requires both five-hour and seven-day windows below 100%. Successful completion also clears it. An explicit model pick bypasses the cooldown for that prompt, including a queued prompt and a model pick persisted before the next send.
Lifetime: cooldowns are process-scoped, expire automatically after at most six hours, and are not restored after a server restart. Only a new failure re-arms one. No database migration. Model limits, invalid/past resets and unknown scopes do not cool an account.
Tests:
ProviderCooldown.test.tscovers the six-hour cap, manual clearing, fresh usable quota recovery, and stale/partial quota rejection. The existing provider-switch table covers queued failover/restart/claimed-run guards, default immediate routing, same-default composer payloads, different explicit picks, scheduled next-turn picks, and deferred preparation/release. All five reproduced routing regressions pass. Earlier registry refresh/stream regressions remain unchanged.Acceptance: after a natural account limit with a future reset, inspect provider status for the bounded cooldown warning. Query C must show default subsequent runs using the fallback without another failed source-instance attempt. Manually refreshing provider status, a fresh usable quota probe, successful completion, or expiry must remove the warning. An explicit pick of the cooling instance must retain that requested instance. The isolated proof can be rerun without calling a real provider:
4. ChatGPT sharing unsupported capability
Evidence: six errors normalize to
subscription_sharing_unsupported_capability. Existing request handling distinguishes unsupported tool namespaces andadditional_tools; the available evidence does not establish an account or plan denial.Outcome: item 4: classification confirmed, notice improved
Change: the run's notice names the actual Codex provider instance and the unsupported capability, either tool namespaces or
additional_tools. It suggests removing the corresponding thread tools or using another provider. Classification remains unchanged, with no automatic account failover for this code and no raw native-error detail in the notice.Notice text:
Codex instance "<instanceId>": ChatGPT sharing does not support tool namespaces. Remove namespaced tools from this thread or use another provider.OrCodex instance "<instanceId>": ChatGPT sharing does not support the additional_tools input item. Remove additional tools from this thread or use another provider.Tests: three rows added to the existing adapter classification table cover namespace terminal errors,
additional_toolsnotifications, and namespace retry notifications. They failed before the fix at the missing instance/capability assertions and now pass while retainingprovider_error, no account-failure eligibility, and sanitized notices. All 19 table scenarios and all 6CodexManagedErrors.test.tscases pass.Acceptance: after a natural post-deploy unsupported-capability error, query D must show the provider instance and the relevant namespace or
additional_toolsguidance. Inspect any new account/plan denial evidence before broadening classification. Reproduce the notice and classification checks without invoking a real provider:5. Codex active-writer conflict
Evidence: two “conversation is open in another Codex client” failures.
Change: resume retries only the recognized active-writer conflict with exponential backoff: 250 ms, 500 ms, 1,000 ms, then fail normally after at most four attempts. Other errors are not retried by this policy.
Tests: the existing Codex resume test now covers recovery and bounded exhaustion under the test clock.
Acceptance: rerun the isolated adapter regression (no real provider invocation):
Read-only post-deploy queries
Set
DEPLOYED_ATto the actual release deployment time in UTC. Do not infer deployment from merge time. All database access below is read-only.No matching natural post-deploy failures means that live acceptance is not yet observed; the isolated tests remain the reproducible proof. These commands do not trigger provider turns or spend money.
Verification and scope
vp run check:changed origin/integration/mcp-events-20261109on rebased head33bf3cd69f31021c834f965479b3e7f05aa883cc: format, lint, knip, GraphQL generation, 37-package affected typecheck and 1,042 tests / 18 files pass.Implemented with Codex CLI.