Skip to content

perf(yearn): query small parent flows on the blockTimestamp index (fix Envio 504s) - #389

Merged
spalen0 merged 2 commits into
mainfrom
fix/small-flows-timestamp-query
Sep 27, 2026
Merged

spalen0 merged 2 commits into
mainfrom
fix/small-flows-timestamp-query

Conversation

@spalen0

@spalen0 spalen0 commented Sep 27, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #388, which added retries. This PR fixes the root cause of the Small parent flow monitor: Envio GraphQL request failed (HTTP Error 504: Gateway Timeout) alerts (4 of 24 hourly runs on Sep 26–27).

Why

Once a stream has a cursor, the query filtered on a blockNumber range with sinceTs=0. The Envio schema only has single-column indexes, and block numbers overlap across chains (katana's position at 43.7M falls inside Base's block range). Postgres walks the global blockNumber index through other chains' rows, and the hosted gateway gives up at about 15s. alert_large_flows filters on blockTimestamp, which is comparable across chains, and never failed.

Live benchmark against production Envio with production cursors (5 runs each):

variant katana deposit mainnet withdrawal slowest overall
current (blockNumber range + _or) median 15.1s, mostly 504 max 15.1s (504) 504
blockNumber range without _or median 15.1s, mostly 504 max 5.6s 504
+ blockTimestamp floor 0.16s 0.16s 0.43s

The retries from #388 don't help much when a query consistently takes more than 15s, as katana deposits currently do.

What changed

  • blockTimestamp >= <cursor block timestamp> now sits at the top level, next to the existing _or, so the cursor semantics stay the same.
  • The cursor state now stores block_timestamp. EventCursor ordering and equality are unchanged, because the new field is excluded from comparison.
  • Existing cursors without a timestamp look it up with a blockNumber equality query (about 0.15s, which is selective). They are upgraded the next time the cursor is saved. If no event is indexed at the cursor block, the monitor logs a warning and falls back to the old unfloored query.

Tests

  • New tests cover the floor placement in the query, saving and loading the cursor timestamp, the floor advancing across pages, the legacy lookup and its fallback, and the lookup query shape.
  • pytest: 1407 passed, 6 skipped. ruff is clean. mypy reports no new errors.
  • Dry run of monitor_flow_type for all chains against production Envio, using a copy of the production state DB (no alerts sent, no state saved): every stream finished within 2.6s. Katana deposits took 0.14s.

🤖 Generated with Claude Code

spalen0 and others added 2 commits September 27, 2026 18:55
…ndex

The cursor query used a blockNumber range with sinceTs=0 once a cursor
existed. Envio only has single-column indexes and block numbers overlap
across chains (katana 43.7M falls inside Base's range), so Postgres walked
other chains' rows and the hosted gateway returned 504 after ~15s. Live
benchmarks: katana deposits 504'd on most attempts; the same query with a
blockTimestamp floor peaked at 0.43s.

Store the cursor block's timestamp and use it as a top-level
blockTimestamp floor. Legacy cursors without a timestamp resolve it once
via a cheap blockNumber equality lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… filter

A cursor from a vault that has since left the active set was excluded by
the vaultAddress filter, so the lookup returned None and every run fell
back to the unfloored query that 504s. Every event in a block shares its
timestamp, so match on chain and block only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@spalen0
spalen0 merged commit 90f9935 into main Sep 27, 2026
3 checks passed
@spalen0
spalen0 deleted the fix/small-flows-timestamp-query branch September 27, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant