Repository navigation
fix(bot): stop retrying a webhook delivery forever when it throws - #313
Merged
Merged
Conversation
A Discord submission webhook whose submission had been deleted fell back to the thin outbox payload. buildDiscordWebhookBody then called truncate() on an undefined status and threw. The poll loop caught the error, but the row never had its attempts counted or its next attempt moved, so it was retried on every tick. One ATLAS delivery had been retried every 15 seconds since Nov 2. - Discord deliveries whose submission no longer exists are marked failed with "Submission no longer exists." Generic JSON hooks keep getting the stored thin payload, as before. - Any exception during a delivery now counts as a failed attempt with backoff, so a poisoned row ends up failed after MAX_ATTEMPTS. - Retry state lives in a pure helper (webhook-retry.ts) with tests.
|
Graphify reviews are paused for this workspace because nobody has used it in 7 days. A workspace owner can resume reviews from the Repositories page: https://app.graphify.com/msk-scripts/repositories |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
One webhook delivery in production had been retried every 15 seconds since Nov 2 (about 3,900 log lines today alone) with
Cannot read properties of undefined (reading 'length').Cause: the submission behind a Discord
submission.createddelivery was deleted before the delivery went out.hydratePayloadthen fell back to the thin outbox payload, andbuildDiscordWebhookBodycalledtruncate()on its undefinedstatus. The exception was caught by the poll loop, but the row never had itsattemptscounted ornextAttemptAtmoved, so it stayed due on every tick.Changes:
apps/bot/src/webhooks.ts: Discord deliveries whose submission no longer exists are markedfailedwith "Submission no longer exists.". Generic JSON hooks still get the stored thin payload, as before. An exception during a delivery now goes throughrecordCrash, which counts it as a failed attempt with backoff.apps/bot/src/webhook-retry.ts:failedAttempt()(attempt count, backoff, give-up atMAX_ATTEMPTS) andbackoffMs, moved out ofwebhooks.tsso they can be tested without a database. The normal failure path uses the same helper.apps/bot/src/webhook-retry.test.ts: tests, including a row that fails on every tick ending upfailedafterMAX_ATTEMPTS.The stuck production row will be marked failed on the first tick after deploy, no manual database change needed.
Type of change
Checklist
pnpm lint,pnpm typecheck,pnpm buildpass locally (bot package: lint, typecheck, tests; the rest is left to CI)attempts, three of the four tests fail.envvalues committedLinked issues
None