feat(core): put a rate-limited queue message back at its retry time

[d-klotz]

A queue handler that hits a provider rate limit no longer gives its message back to SQS for a 30 minute redelivery that spends one of its three receives. The queue worker puts the message back so it runs at the time the provider said. This is phase 2 of ADR-049 (#655).

Stacked on #656. Merge #656 first. This PR needs RateLimitError and retryAt from it, and GitHub retargets this PR to next when #656 merges.

What problem this fixes

• A long limit costs all three receives. After the Requester's in-process ladder, a throttled handler rethrows and SQS redelivers 30 minutes later. The queue allows 3 deliveries, so a limit that lasts two hours sends the message to the DLQ, which only logs.
• A sync split into many messages then stays open forever. One lost message means the run is never completed and never failed.
• The user sees a run that looks stuck. Nothing says "waiting for the provider limit until 23:00 UTC".

What this PR does

When a handler throws an error with isRateLimited and a retryAt (a RateLimitError from the Requester, or one a module throws), Worker.run puts the message back so it runs at retryAt:

Wait	What the worker does	Record
Up to 900 s	Sends the same body again with DelaySeconds, then acknowledges the old message	handled
Over 900 s, SCHEDULER_ROLE_ARN set	Creates a one-time EventBridge schedule at retryAt that sends the body to the queue, then acknowledges	handled
Over 900 s, no scheduler	Extends the visibility timeout to retryAt (12 h less 20 minutes at most: SQS counts the 12 h from the receive, and an extension does not reset it). This uses one receive	reported failed

• The new message goes out before the old one is acknowledged. Delivery stays at least once, as today. A crash between the two can run the work twice, so handlers stay idempotent.

• A deferred body carries _frigg: { deferrals, firstDeferredAt, deferralId }. It is reserved; handlers never see it. deferralId is the SQS id of the first message that was put back, and later deferrals keep it. The new message starts at receive count 1.

• Two caps bound it: FRIGG_QUEUE_MAX_DEFERRALS (default 30) and FRIGG_QUEUE_MAX_DEFERRED_MS (default 26 h, from the first deferral to the retry time). The HubSpot and Salesforce modules probe a daily limit every hour, so the defaults cover one full day of hourly probes. Core owns both ends of these numbers, so devtools injects nothing.

• Past a cap, with no retryAt, with no event source ARN, or on a FIFO queue, the record fails like any retryable error does, and delivery.isLastAttempt (feat(core): tell queue handlers their last delivery attempt #653) applies.

• Run state. With a processId in the message, the worker writes, through two use cases (RecordRateLimitWait, ClearRateLimitWait), Process.context.rateLimit = { status: 'WAITING' | 'EXHAUSTED', mechanism, retryAt, reason, module, deferrals, deferralId, updatedAt }. After a deferred or redelivered message ran, a WAITING value becomes null at once when that message set it (same deferralId), and for any other message only when its retryAt is past or within 5 s. So a job that resumes before its retryAt clears its own wait, and cannot clear another message's wait. applyProcessUpdate also keeps the process updatedAt fresh while it waits.

• A throttled token endpoint defers too. OAuth2Requester replaces non-definitive refresh errors with a fresh Error on purpose (the original message can hold the client secret). The fresh error now keeps isRateLimited, retryAt, waitMs, reason and module, none of which is secret.

• Devtools. The one-time scheduler is provisioned when any API module declares static rateLimit. The base role gains sqs:ChangeMessageVisibility.

• One log record per outcome. Acknowledged: WARN record_deferred. Not acknowledged before the last receive: one WARN (record_deferral_capped, record_visibility_extended, or record_failed with deferral.skipped and the cause as an error). Not acknowledged on the last receive: one ERROR record_lost_rate_limited with the same details, and no WARN.

• A scheduler that cannot be built is handled like a failed schedule: the record falls back to the visibility tier with scheduleError, instead of failing the whole batch.

• The scheduler client honours the local endpoint. The EventBridge adapter now takes the same awsConfigOptions() as the SQS clients. Before, a local run with SCHEDULER_ROLE_ARN set sent a long deferral to the real AWS endpoint with the machine's own credentials; the local end-to-end run found it.

Handler contract

Check the last attempt first, then rethrow:

async processBatch({ data, delivery }) {
    try { await this.syncPage(data); }
    catch (error) {
        if (delivery?.isLastAttempt) return this.failRun(data.processId, error);
        throw error; // a RateLimitError too: core puts the message back at retryAt
    }
}

The delay and schedule tiers send a new message, so they never reach the last attempt. The visibility tier and a cap use a receive. On the last receive the worker writes an ERROR, frigg.worker.record_lost_rate_limited, and the handler has that one chance to end the run. (ADR-049 Appendix A.10 rethrows a rate-limited error first; that order leaves the run open in those two cases, so the check moves first.)

Design choices worth a look

• The mock scheduler is never used for a deferral. The factory returns it for STAGE=dev, and a schedule kept in memory would be lost with the acknowledged message. The worker builds the EventBridge adapter explicitly, and only when SCHEDULER_ROLE_ARN is set and SCHEDULER_PROVIDER is not mock.
• The queue URL comes from record.eventSourceARN (GetQueueUrl by name, memoized per invocation). The base Worker has no integration name for a <NAME>_QUEUE_URL, and a composed URL breaks LocalStack and other partitions.
• The Worker's SQS client now takes the offline and endpoint options that QueuerUtil has. Deferral puts GetQueueUrl, SendMessage and ChangeMessageVisibility on the normal path; without it, frigg start and LocalStack would reach real AWS on every deferral.
• sqs:ChangeMessageVisibility is granted to every stack, not only to one with a declaring module. feat(core): honour Retry-After and let API modules declare rate limits #656 honours Retry-After for every module, so any queue worker can meet a long wait. The statement is already scoped to <service>--<stage>-*Queue; the template gains one action string, and no new resource, function or environment variable. ADR-049 §8 asks for a byte-identical template when no module declares rateLimit; this is the one deviation.
• @aws-sdk/client-scheduler is now a declared dependency of core. Core already required it (the scheduler adapter is exported from the package root) and got it only because admin-scripts hoists it. The lockfile gains one line; npm ci --dry-run passes.

Caveats

• Not run against real AWS. Every tier ran end to end against LocalStack, and is unit-tested with aws-sdk-client-mock. Before relying on it in production, run one throttled message per tier on a dev stack and check the queue, the schedule group and CloudTrail.
• ReportBatchItemFailures and our own ChangeMessageVisibility. The visibility tier reports the record failed after extending its timeout. As far as I know Lambda does not reset the visibility of a reported failure, but I have not verified it. If it does, that tier falls back to today's 30 minute redelivery.
• A stack whose queue is not its own (ownership.queue: 'external') needs sqs:ChangeMessageVisibility and scheduler access granted separately. Pre-existing and wider now: createSchedulerExecutionRole names <Name>Queue for every integration, so such an app cannot deploy once the scheduler is on.
• A wait over 26 hours (a monthly quota) is capped at once and falls back to delivery.isLastAttempt. Raise FRIGG_QUEUE_MAX_DEFERRED_MS for an app that wants longer.
• The handler contract ends a run on the last receive, also when core could still defer. A handler that checks delivery.isLastAttempt first ends the run and returns, so on the third receive a RateLimitError that the delay or schedule tier could still put back ends the run instead. A message reaches its third receive only after two failures that were not deferred, so this is rare.
• The run state is read, then written. Two workers that write in the same instant can keep the earlier retryAt. It is a UI signal only, and the next write corrects it.
• Twenty workers on one limit each defer their own copy. Nothing paces them yet.

Not in this PR

The RATE_LIMITED message for the user, pacing, and the policies of the HubSpot, Salesforce and Pipedrive modules.

Test plan

• core: npm test in packages/core — 2,817 pass (with feat(core): honour Retry-After and let API modules declare rate limits #656 merged in); the 63 failures (DocumentDB, Prisma decryption, Mongo memory server) are the same 63 that fail on origin/next in this environment
• core/Worker.deferral.test.js: 44 tests — the three tiers, the two caps, the 12 h margin, every skip reason, one log record per outcome also on the last receive, a scheduler that cannot be built, the hooks, the SQS client options
• queues/queue-deferral.test.js: 22 tests. oauth-2.test.js: 2 new. integration-defined-workers.test.js: the hook wiring and the clear after a deferred run; the merge rules in record-rate-limit-wait.test.js and clear-rate-limit-wait.test.js
• devtools: 9 new tests (scheduler switch, IAM action, and a template with no declared rate limit gains no scheduler resource, function or env var); the 36 failures in infrastructure/domains are the same on origin/next
• aws-sdk-client-mock is now a declared dev dependency of core: one lockfile line, npm ci --dry-run passes, one copy
• type declarations checked with tsc --strict against a Worker subclass that overrides the hooks
• local end-to-end run (frigg-local-test harness: real Postgres, LocalStack SQS and Scheduler, a real QueueWorker from createQueueWorker, and an integration whose API module calls a local fake provider, with the feat(core): honour Retry-After and let API modules declare rate limits #656, feat(core): put a rate-limited queue message back at its retry time #657 and feat(core): tell the integration when a rate limit is too long to wait for #658 core combined):
  • the delay tier sends a new message with _frigg.deferrals: 1 that arrives 3 s later as receive 1, and the run state goes from WAITING to null after it runs
  • the visibility tier hides the message with a real ChangeMessageVisibility
  • the schedule tier creates frigg-defer-<messageId> with an at(...) time, the queue as target and the deferred body, and acknowledges the record
  • an earlier wait does not replace a later one in the run state
• local full-app run (frigg start on LocalStack SQS and Scheduler and Postgres; 4 fake API-key modules that throttle like HubSpot, Salesforce, Quo and Clockwork; the Management API creates the integrations and starts 6-page syncs; feat(core): honour Retry-After and let API modules declare rate limits #656, feat(core): put a rate-limited queue message back at its retry time #657 and feat(core): tell the integration when a rate limit is too long to wait for #658 combined): 29/29 checks. It covers in-process waits, the 5 min cap, the delay and schedule tiers (the runner fires due schedules, because LocalStack 3.8.1 does not), one warning per long wait, each page fetched exactly once, and a run state that is cleared also when a job resumes 60 min before its retryAt
• eslint: 0 errors. Warnings per file match the base, except 4 camelcase warnings in oauth-2.test.js on the grant_type and refresh_token constructor params, which the 58 existing warnings in that file share
• CI build (22.x) fails on feat(core): add queryMappings to IntegrationMappingRepository #651, feat(core): tell queue handlers their last delivery attempt #653 and feat(core): structured logging (ADR-048) #654 too (frigg-cli, devtools infrastructure and management-ui suites, and a missing libcrypto.so.1.1 for the Mongo memory server), so it does not gate this PR

Release notes

• Upgrade @friggframework/core and @friggframework/devtools together. Devtools grants the IAM action and switches on the scheduler; core does the deferral.
• Redeploy the stack for the IAM action and, when a module declares rateLimit, for the scheduler resources.

🤖 Generated with Claude Code

📦 Published PR as canary version: 2.0.0--canary.657.07aea31.0

✨ Test out this PR locally via:

npm install @friggframework/admin-scripts@2.0.0--canary.657.07aea31.0
npm install @friggframework/core@2.0.0--canary.657.07aea31.0
npm install @friggframework/devtools@2.0.0--canary.657.07aea31.0
npm install @friggframework/eslint-config@2.0.0--canary.657.07aea31.0
npm install @friggframework/prettier-config@2.0.0--canary.657.07aea31.0
npm install @friggframework/schemas@2.0.0--canary.657.07aea31.0
npm install @friggframework/serverless-plugin@2.0.0--canary.657.07aea31.0
npm install @friggframework/test@2.0.0--canary.657.07aea31.0
npm install @friggframework/ui@2.0.0--canary.657.07aea31.0
# or 
yarn add @friggframework/admin-scripts@2.0.0--canary.657.07aea31.0
yarn add @friggframework/core@2.0.0--canary.657.07aea31.0
yarn add @friggframework/devtools@2.0.0--canary.657.07aea31.0
yarn add @friggframework/eslint-config@2.0.0--canary.657.07aea31.0
yarn add @friggframework/prettier-config@2.0.0--canary.657.07aea31.0
yarn add @friggframework/schemas@2.0.0--canary.657.07aea31.0
yarn add @friggframework/serverless-plugin@2.0.0--canary.657.07aea31.0
yarn add @friggframework/test@2.0.0--canary.657.07aea31.0
yarn add @friggframework/ui@2.0.0--canary.657.07aea31.0

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-28T23:44:40.707723Z	ebe5c60	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ebe5c602d3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve rate-limit waits from other messages

When a process has multiple queued messages, this clears the process-wide context.rateLimit after any successful message carrying _frigg, or even after an unrelated ordinary retry whose receive count exceeds 1. If another message for the same process is still deferred, clearRateLimitWait() consequently sets the shared state to null and the run no longer appears rate-limited even though work is still waiting. Track the deferred message in the stored state and clear it conditionally, or maintain aggregate per-message waits.

Useful? React with 👍 / 👎.

[d-klotz]

Addressed the automated review in the latest commit:

• Shared context.rateLimit (Codex P2): the value is one per process, so an unrelated message that succeeded could clear it while another message still waited. The queue worker now compares times. A new wait replaces the stored one only when its retryAt is later, a WAITING value never hides an EXHAUSTED one, and after a deferred or redelivered message ran the worker clears the value only when it is WAITING and its retryAt is past or within 5 s (the scheduler fires up to a second early). Both hooks now read the process first; a failed read is logged and never changes the outcome. 15 tests cover it.
• SonarCloud: defer() is split into one helper per tier, which brings its cognitive complexity under the limit. The two no-op hooks have a body. The redelivery check no longer negates a comparison, because !(receiveCount > 1) is not receiveCount <= 1 when the count is unknown.

This branch also merges the fixes made to #656 (the review of the Requester), so the stack stays in sync.

Full packages/core run: 2,769 pass, and the same 63 failures as origin/next in this environment.

[claude]

Beyond the inline findings, I also checked two other candidate issues and ruled them out: context.rateLimit (backend-utils.js) is a brand-new field with no prior usage anywhere in the codebase, so there's no overwrite risk for existing integration data; and the visibility-extension deferral path for externally-owned/non-scheduler queues is still bounded by SQS's native receive-count/isLastAttempt check (independent of the body-embedded deferral counter), so it can't extend visibility forever even though that counter itself doesn't advance on redelivery.

Extended reasoning...

This PR is a large, security/reliability-sensitive change to the queue worker's rate-limit deferral logic (Worker.js, backend-utils.js, oauth-2.js, scheduler-builder.js) with three confirmed findings already flagged inline (a scheduler IAM/deploy gap, a lost module name on OAuth rate limits, and a rate-limit-state clearing race across sibling messages). I independently traced two additional candidate concerns from the ruled-out list — a potential field-overwrite in backend-utils.js and a potential unbounded-visibility-extension loop for external queues in Worker.js — and confirmed via the code (grep for prior context.rateLimit usage, and the delivery.isLastAttempt gate at Worker.js's visibility branch) that neither holds up as a bug.

Findings marked 🟡 are optional suggestions and need no follow-up push.

[claude]

🔴 Deploys can now fail for apps that have no webhooks at all: any module anywhere declaring static rateLimit turns the scheduler on via declaresRateLimit (scheduler-builder.js:22-25, used at line 47), and createSchedulerExecutionRole (unchanged, line 118-122) still emits Fn::GetAtt: ['${Name}Queue','Arn'] for every integration regardless of queue ownership. If any integration in the app has ownership.queue: 'external' (its queue isn't created as a <Name>Queue resource in this stack, per integration-builder.js:227-229), that GetAtt can't resolve and the whole stack fails to deploy once any module declares rateLimit -- even in integrations unrelated to that queue. …

Why this was flagged

…Before this PR only webhooks triggered the scheduler, so this stack-wide failure only affected webhook-enabled apps with an external queue; now it also hits non-webhook apps as soon as one module opts into rate-limit handling. Fix: have createSchedulerExecutionRole skip or use an ARN reference (not Fn::GetAtt) for any integration whose queue ownership is external, on every path that can enable the scheduler (webhooks and rateLimit).

Trigger: an app defines integration A whose queue ownership is 'external' (no webhooks) and integration B whose module declares static rateLimit. Entry point: serverless build calls SchedulerBuilder.shouldExecute(appDefinition), which now returns true via declaresRateLimit(integration) at scheduler-builder.js:47, then build() calls createSchedulerExecutionRole (scheduler-builder.js:110-166) which pushes {'Fn::GetAtt': ['AQueue','Arn']} at line 122 for integration A even though its queue was never created as a stack resource (integration-builder.js:227-229 only creates it when ownership is STACK). CloudFormation cannot resolve that GetAtt, so `serverless…

Verification: normal; acknowledged in diff (the note's claim holds). The change adds declaresRateLimit (scheduler-builder.js:22-25) into shouldExecute at line 47, so the scheduler now switches ON for an app that merely has a module declaring static rateLimit, with no webhooks and no scheduler.enable. Once ON, build() calls createSchedulerExecutionRole (unchanged, lines 111-172), which at lines 114-125…

[claude]

🟡 (optional) Operators lose the module name in Process.context.rateLimit when a token-refresh call is rate limited, unlike a normal request throttle. _transportFailureError (oauth-2.js:392-407) copies isRateLimited, retryAt, waitMs and reason from the RateLimitError onto the fresh transportError but never copies error.module, even though moduleName is already computed at oauth-2.js:352 for the error message. Worker.defer reads error.module (Worker.js) into details.module and into recordRateLimitWait's written context.rateLimit.module (backend-utils.js:180), so it comes out undefined for this path even though Requester.js:373 sets module on every other RateLimitError. Fix: also set transportError.module = error.module ?? …

Why this was flagged

…moduleName in _transportFailureError so the run-state visibility this PR adds names the throttled module consistently for both paths.

When a provider's OAuth token endpoint returns 429/5xx during refresh, OAuth2Requester.refreshAuth (oauth-2.js:349-368) calls _transportFailureError(error, moduleName) and rethrows the returned transportError. That error's isRateLimited/retryAt/waitMs/reason are copied but .module is not (oauth-2.js:401-406). Worker.defer (Worker.js) then reads error.module for the log details and for the Process.context.rateLimit.module field written via recordRateLimitWait (backend-utils.js:171-186), so operators/users see rateLimit.module = undefined for this trigger, unlike the module name shown for requests throttled directly by Requester.js (which sets module at line 373). This weakens the PR's own stated goal that the run state should say what is waiting.

Verification: nit. Real omission, but observability-only (no functional break). _transportFailureError (oauth-2.js:392-407) copies isRateLimited/retryAt/waitMs/reason from the original RateLimitError onto the fresh transportError but never copies .module: ``` if (error?.isRateLimited) { transportError.isRateLimited = true; transportError.retryAt = error.retryAt; transportError.waitMs =… | nit (new…

[claude]

🟡 (optional) An operator's Process.context.rateLimit wait status can silently vanish while a rate limit is still active, because clearing it is keyed only on the current message's own retry count, not on which message set it. Worker._clearDeferredState (Worker.js:332-336) calls clearRateLimitWait whenever this message's own receiveCount>1, and backend-utils.js:186-192 does an unconditional applyProcessUpdate($set 'context.rateLimit'=null) by processId only. Since one process's sync commonly queues many messages sharing a processId, an unrelated message that fails once for any reason and later succeeds wipes the WAITING/EXHAUSTED status a different, still-pending, rate-limited message for that process legitimately set. …

Why this was flagged

…Fix: only clear when this message itself carries deferral state (_frigg.deferrals>0), or make the clear conditional on the stored rateLimit record matching this message/mechanism instead of a blind $set.

Process P queues messages A and B sharing one processId (PR's own 'sync split into many messages' pattern). A is rate-limited and defer()'d, so recordRateLimitWait (backend-utils.js:168-184) sets Process.context.rateLimit={status:'WAITING',...}. B, unrelated to rate limiting, fails once for any transient reason, is redelivered (receiveCount=2), then succeeds; Worker._clearDeferredState (Worker.js:332-336) sees receiveCount>1 and calls clearRateLimitWait, which backend-utils.js:186-192 runs as an unconditional $set 'context.rateLimit'=null. This erases A's still-valid WAITING/EXHAUSTED record even though A has not run yet. The base branch has no such field; after merge the operator-visible wait status can disappear while the limit is still active, with no check on message identity or mechanism before clearing.

Verification: nit (observability-only). Real, reachable logic defect newly introduced by this PR, but with no functional/delivery/data impact. Worker.js:332-336 (_clearDeferredState): the guard if (readDeferral(body).deferrals === 0 && !(delivery.receiveCount > 1)) return; means the process-wide clear runs whenever the CURRENT message has deferrals > 0 OR receiveCount > 1. The receiveCount > 1…

[claude]

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.

[claude]

⚠️ Code review skipped — your organization has no extra usage available to pay for this review.

If your organization's extra usage balance is empty, an organization admin can add extra usage credits at claude.ai/admin-settings/usage. If its monthly spend limit was reached, an admin can raise it on the same page. If neither applies, contact Anthropic support.

Once extra usage is available, comment @claude review on this pull request, or push a new commit, to trigger a review.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud