Skip to content

fix(server): recover Claude cooldown reset times from proxy errors - #36642

Merged
juliusmarminge merged 14 commits into
mainfrom
factory/pr-26735-a1794649826
Nov 14, 2026
Merged

juliusmarminge merged 14 commits into
mainfrom
factory/pr-26735-a1794649826

Conversation

@juliusmarminge

@juliusmarminge juliusmarminge commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Nothing needed changing. PR #36642 at 1f88524: all 35 checks passed (8 of them required). Macroscope approved it with no line comments. CodeRabbit's review says 'No actionable comments were generated' and rates the merge risk Minimal. No reviewer finding is open. The fix itself is the same as the last round: the adapter reads the CLIProxyAPI cooldown only from the outer 'All credentials … are cooling down (reset after …)' message and ignores nested 'last error' reset hints. It combines that deadline with native Claude rate-limit resets: the latest known reset wins, and an unknown native reset stays unknown.

Supersedes #26735. The change is @greyxp1's; their commits are kept.

Reproduction on main

Reproduced on main at 0fee26d:

  1. Started a local stand-in proxy that answers each request with HTTP 429 and error message All credentials for model claude-opus-5-5 are cooling down (last error: rate_limit_error (reset after 1h)). Running the claude CLI directly against it produced API Error: Request rejected (429) · All credentials for model claude-opus-5-5 are cooling down (last error: rate_limit_error (reset after 1h)), exactly the PR's example. 2) Started the main dev server on a fresh database with the Claude provider pointed at the stand-in, and turned on 'Resume after usage limits' in Settings → Threads. 3) Added a demo project and started an Opus 5.5 thread with 'Say hi'. 4) The turn failed at 09:56:43Z. The persisted failure is {class: usage_limit, code: api_error_429, resetAt: 2026-11-14T10:56:43.553Z}, and the thread's limitRecovery has the same resetAt with autoResume: true. The UI shows 'Limited' and 'Usage limit reached · Resumes 10:56 AM'. That one-hour deadline comes only from the nested last-error text; the proxy's own message has no cooldown, so the reset should be unknown. This reproduces the first defect. 5) Second scenario (outer proxy cooldown together with a native rejected rate_limit_event): with the proxy also returning anthropic-ratelimit-unified-* rejected headers, the API-key CLI still emits no rate_limit_event (main's code notes that API-key sessions get no rate_limit_info). Native limit events need first-party Claude OAuth, which no VM has, so the 'later deadline discarded' and 'unknown native reset replaced' cases were not run end to end.

Evidence

Run the real Claude CLI through the real server and web client against an HTTP 429 stand-in whose only deadline is nested inside last error: rate_limit_error (reset after 1h). With automatic recovery enabled, open a fresh Opus 5.5 thread and send Say hi. Both sides use fresh synthetic state, Chromium on Linux, light theme, and a 1280×800 CSS viewport.

Before After
13f89ae0a7 1f88524e9d
Before: nested cooldown incorrectly produces Resumes 11:18 AM After: reset time remains unavailable and requires manual retry
Before recording After recording

Base persisted a one-hour reset deadline and armed automatic recovery. Head persisted resetAt: null and no recovery schedule. Both received the same proxy error.

Proxy script · Seed script · Reproduction instructions · Observed persisted state

Not shown:

  • Permanent blocker: Combining a proxy cooldown with a native blocking rate_limit_event requires first-party Claude OAuth, unavailable on any VM. The later-proxy-deadline and unknown-native-reset cases were not run end to end.
  • The one-hour recovery timer was not allowed to expire; evidence covers the displayed deadline and persisted recovery schedule.

Tests

No new local runs this round, because no code changed. On the PR, CI passed all 35 checks on 1f88524. Last round: vp test run ClaudeAdapterV2.test.ts -t 'recovers wrapped gateway limits' passed 13 of 13, and vp run check:changed passed.

Not in this change

  • Automatic recovery when every proxy account is exhausted is not tested end to end (no first-party Claude OAuth or proxy pool on the VM).
  • CodeRabbit notes the PR description's Evidence section is still marked as being captured; the pipeline fills in the PR description, so I left it.

Who did the work

  • Written by Claude Opus 5.5, in Claude Code
  • Reviewed by GPT-6.1 Sol, in Codex, and by Claude Opus 5.5, each on its own
  • Proof captured by GPT-6.1 Sol and checked by Claude Opus 5.5

Run by the t3code factory in T3 Code.

@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Nov 14, 2026
@macroscopeapp

macroscopeapp Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 1f88524

Macroscope's review found this PR approvable — This change recovers proxy cooldown timestamps while avoiding nested upstream reset hints and preserving unknown-limit handling. Its effects are limited to existing usage-limit reporting and recovery timing.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ The exact PR base did not have a successful artifact. Baseline uses the latest successful main measurement shown below.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

All 64 scenario measurements match main.

All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded 10.5 KiB 10.5 KiB 0 B (0.0%) 12.7 KiB ✅
Codex short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn running, cold open HTTP decoded 281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅
Codex long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Codex long turn running, history page HTTP decoded 92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅
Codex long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Codex long turn finished, cold open HTTP decoded 381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, socket cold open WebSocket decoded 381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, work page HTTP decoded 186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅
Codex long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Codex idle thread resume WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell snapshot HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Codex fleet shell subscribe WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Codex fleet shell resume check rows 0 0 0 0 ✅
Codex stale shell reload HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex stale shell reload rows 27 27 0 (0.0%) 27 ✅
Codex stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Codex stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex stale todo list cold open HTTP decoded 72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅
Codex stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Codex stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅
Claude short turn resume WebSocket decoded 10.6 KiB 10.6 KiB 0 B (0.0%) 12.7 KiB ✅
Claude short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn running, cold open HTTP decoded 283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅
Claude long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Claude long turn running, history page HTTP decoded 93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅
Claude long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Claude long turn finished, cold open HTTP decoded 384.2 KiB 384.2 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, socket cold open WebSocket decoded 384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, work page HTTP decoded 188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅
Claude long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Claude idle thread resume WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell snapshot HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Claude fleet shell subscribe WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Claude fleet shell resume check rows 0 0 0 0 ✅
Claude stale shell reload HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude stale shell reload rows 27 27 0 (0.0%) 27 ✅
Claude stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Claude stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude stale todo list cold open HTTP decoded 73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅
Claude stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Claude stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅

Baseline: 0fee26d · PR result: 1f88524 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The adapter now parses CLIProxyAPI cooldown text for API 429 failures. During turn finalization, it combines SDK reset times with usage-limit failure reset times and applies the latest time to the terminal usage-limit failure. Tests cover proxy responses, reset values, and failure classification.

Changes: Proxy rate-limit handling

Layer / File(s) Summary
Parse proxy cooldown responses
apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts, apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts
API 429 failures that match CLIProxyAPI’s “Request rejected” and “All credentials” text use a dedicated reset pattern. Tests cover proxy message variants and classify proxy cases as usage_limit.
Combine usage-limit reset times
apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts, apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts
Turn finalization includes a non-null reset time from a usage-limit result when selecting the latest reset. Tests cover SDK reset events and resetAt values for proxy and non-proxy cases.

Estimated code review effort: 3 (Moderate) | ~15 minutes

Change: Bug fix

Suggested reviewers: t3dotgg, yash-singh1, maria-rcks

Merge Risk: Minimal · up to 1f885

This change improves how Claude rate-limit cooldown deadlines are recovered from proxy errors. It is narrowly scoped and has targeted tests. No merge-blocking risk was identified.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: the Claude adapter now recovers cooldown reset times from CLIProxyAPI 429 errors. The changeset matches: it parses the outer 'all credentials cooling down' message, ig…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description follows the template's sections (Problem, Change, Scope, Verification) and gives specific reproduction steps, before/after evidence, and test results. It also states that the author's …
Full details: Title check

Explanation

The title names the main change: the Claude adapter now recovers cooldown reset times from CLIProxyAPI 429 errors. The changeset matches: it parses the outer 'all credentials cooling down' message, ignores nested 'reset after' text, and combines proxy and native reset times. The title uses the conventional commit format and is specific, so it passes.

Full details: Description check

Explanation

The description follows the template's sections (Problem, Change, Scope, Verification) and gives specific reproduction steps, before/after evidence, and test results. It also states that the author's commits are kept and that the work is attributed to the model and harness. Required content is present, although the Problem section is not labeled as such and the Scope section does not clearly explain the bug-fix scope beyond the supersession note. The description is mostly complete and substantive.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 13f89ae and 1f88524.

Files selected for processing: 2

  • apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts
  • apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 5 remain after this review.

@juliusmarminge
juliusmarminge merged commit 487733b into main Nov 14, 2026
36 checks passed
@juliusmarminge
juliusmarminge deleted the factory/pr-26735-a1794649826 branch November 14, 2026 10:27
AIdoesmyjob pushed a commit to AIdoesmyjob/t3code that referenced this pull request Nov 14, 2026
…ingdotgg#36642)

Co-authored-by: greyxp1 <greyxp999@gmail.com>
Co-authored-by: Grey <120752906+greyxp1@users.noreply.github.com>
Co-authored-by: maria-rcks <maria@kuuro.net>
(cherry picked from commit 487733b)
kylebegeman pushed a commit to kylebegeman/loom that referenced this pull request Nov 14, 2026
…5 (Running the fork's tests in apps/server)

Failing check: Running the fork's tests in apps/server

**What failed:** In "Running the fork's tests in apps/server", three cases in `ClaudeAdapterV2.test.ts` failed: `preserves Switchboard recovery timing for over thirty days`, `for other model` and `for trailing text`. Each expected a reset time and got `null`.

**Why:** Those cases are part of the fork's `fork: switchboard` test block. They cover proxy messages that the fork's own parser (`switchboardCooldownResetAt`) rejects on purpose: a cooldown over 30 days, a different model, or text after the reset hint. The fork never read a reset from those messages. It relied on upstream's old catch-all `(reset after …)` match, which found the hint anywhere in a 429 message, and the test comment said so. Upstream's pingdotgg#36642 ("recover Claude cooldown reset times from proxy errors") narrowed that for CLIProxyAPI's "All credentials for model … are cooling down" messages. Now only a cooldown directly after "cooling down" counts, so nested upstream errors are ignored. The fork's message form ("cooling down via provider claude (last error: …) (reset after Ns)") has text in between, so upstream no longer finds a reset there. The fork's parser still rejects them too, so the result is `null`. That is upstream's intended behavior, and upstream's code wins.

**What I changed:** Only the fork seam rows in `apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts`. The three rows now expect no reset (`expected: 0`), and the comment above them describes upstream's new reading. I didn't touch the fork's parser or any upstream code, and every other row in the block still asserts what it did before. `docs/fork/seams.tsv` needs no change because the marker count is the same. Rerunning the full `ClaudeAdapterV2.test.ts` gives 449 of 449 passing, and `check_seams` printed nothing.
Pipyakas pushed a commit to Pipyakas/t3code that referenced this pull request Nov 15, 2026
…ingdotgg#36642)

Co-authored-by: greyxp1 <greyxp999@gmail.com>
Co-authored-by: Grey <120752906+greyxp1@users.noreply.github.com>
Co-authored-by: maria-rcks <maria@kuuro.net>
(cherry picked from commit 487733b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 changed lines (additions + deletions). vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants