Skip to content

fix(yearn): retry Envio requests with a 10s per-attempt timeout - #388

Merged
spalen0 merged 1 commit into
mainfrom
fix/envio-request-retry
Sep 27, 2026
Merged

spalen0 merged 1 commit into
mainfrom
fix/envio-request-retry

Conversation

@spalen0

@spalen0 spalen0 commented Sep 27, 2026

Copy link
Copy Markdown
Collaborator

Why

yearn-alert-small-parent-flows sent Envio GraphQL request failed (HTTP Error 504: Gateway Timeout) alerts on 4 of 24 hourly runs (Sep 26 21:05 and 22:05, Sep 27 17:05 and 18:05 UTC). The hosted Envio indexer returned a gateway timeout after 18–36s, and a single failed request skipped the whole run. When I re-ran the same queries with the production saved positions, they finished in 0.1–1.4s, so these are short upstream slowdowns, not a query problem.

What changed

  • alert_small_parent_flows.py and alert_large_flows.py now send Envio requests through utils.http_client.request_with_retry. It retries 5xx errors, 429s, timeouts and connection errors with exponential backoff. The retry work in fix(http): add retry logic to remaining HTTP requests #381 missed these two files because they used urllib directly.
  • Each attempt now times out after 10s instead of 30s, so a fully failed run gives up after about 47s.
  • Small-flow alerts show a short reason (HTTP 504 Gateway Timeout, timed out after 10s) rather than the raw exception, which would include the Envio endpoint URL.
  • Large-flow alerts now also cover timeouts and connection errors. Before, those crashed the script.

Tests

  • Added tests for retrying a 504 with the 10s timeout, and for a single alert without the URL when every retry fails.
  • pytest: 1401 passed, 6 skipped. ruff check is clean. mypy reports no new errors.
  • Ran a real query through both scripts against the production Envio endpoint and both got data back.

🤖 Generated with Claude Code

The small parent flow and large flow monitors called Envio via urllib with
no retry, so a single transient 504 from the hosted indexer skipped the run
and sent an alert (4 of 24 runs on Sep 26-27). Route both through
request_with_retry (retries 5xx/429/timeouts with backoff) and cut the
per-attempt timeout from 30s to 10s.

Small-flow failure alerts now report a short status instead of the raw
exception, which would include the Envio endpoint URL. Large-flow now also
handles timeouts and connection errors instead of crashing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@spalen0
spalen0 merged commit 83a68dd into main Sep 27, 2026
3 checks passed
@spalen0
spalen0 deleted the fix/envio-request-retry branch September 27, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant