Skip to content

Fix: fail fast on rolling provider usage limits #112

Description

@byapparov

Context

A production code-review run with CLI 0.4.3 and zai-coding-plan/glm-5.3 reached the provider’s five-hour rolling usage limit. The CLI treated that response as retryable, made ten retry attempts, and remained alive until the executor's two-hour deadline.

Problem / Goal

A provider response that says its usage limit will reset hours later cannot be recovered within a normal task-execution deadline. Retrying it keeps executions appearing to run while consuming capacity, then reports a timeout instead of the actionable provider limit.

Reproduction

  1. Configure a provider response that is marked retryable and says Usage limit reached for 5 hour with a reset time beyond the task deadline.
  2. Run aictrl run --format json.
  3. Observe the current session retry loop accepts the long delay and retries until the outer executor timeout.
  4. Expected: classify this rolling usage-limit response as terminal, emit the normal structured failure lifecycle, and exit non-zero without sleeping/retrying.

Environment

  • CLI: @aictrl/cli 0.4.3
  • Provider/model: Z.ai GLM-5.3 Coding Plan
  • Downstream executor deadline: two hours

Proposed Approach

In src/session/retry.ts, recognize provider usage-limit/reset responses as non-retryable before the retry delay is accepted. Preserve ordinary transient rate-limit retries, and keep the terminal session_error / invocation completion events and error text available to headless consumers.

Acceptance Criteria

  • A response containing Usage limit reached and a future reset is not retried.
  • Headless JSON mode emits a structured terminal failure and exits non-zero for that response.
  • Normal short-lived 429/rate-limit and overload retry behavior remains covered by regression tests.
  • The error makes clear that the provider usage limit, rather than the executor deadline, caused the failure.

Out of Scope

  • Changing retry policy for all provider errors.
  • Product dashboard/heartbeat instrumentation (handled downstream).
  • Retrying automatically after the provider reset time.

Roadmap Alignment

  • Pillar: EXEC
  • Priority: P1 — production reliability and operator diagnosis.
  • Milestone: Enterprise Observability

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions