Skip to content

fix(llm): ground Yearn V3 strategy alerts in on-chain vault context - #383

Open
spalen0 wants to merge 2 commits into
mainfrom
fix/timelock-explainer-yearn-v3
Open

spalen0 wants to merge 2 commits into
mainfrom
fix/timelock-explainer-yearn-v3

Conversation

@spalen0

@spalen0 spalen0 commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The Yearn timelock alert for 0x2714…d788 (add_strategy + update_max_debt_for_strategy on two WETH vaults) produced this summary:

Adds a Yearn V3 strategy and an Aave looper to separate WETH vaults. Sets raw maximum-debt values of 500000000000000000000 and 10000000000000000000000 (1e18-normalized: 500 and 10,000; units unconfirmed)…

What went wrong, and why:

Symptom Root cause
Neither vault named; both labeled "Yearn V3 Vault" Every V3 vault is an EIP-1167 clone. fetch_erc20_metadata only resolved EIP-1967/getter proxies, so clones got no symbol/decimals/name.
"Aave looper" (it's the wstETH/WETH Spark Looper) The label was the Etherscan contract name LSTAaveLooper. The on-chain name() was never read.
"units unconfirmed" max_debt is always in the vault asset, but nothing said so. Related Tokens listed several candidates (shares, asset(), accountant()).
"meaning of the boolean … not established" add_strategy is overloaded (a Vyper default arg). The param-name lookup took the first ABI entry, add_strategy(address), so add_to_queue was unnamed. Vyper source context was also never extracted (the regex only knew Solidity function).
"debt-setting calls reverted / not verified" The Tenderly bundle request failed on the runner, and the code fell back to simulating each call on its own. Out of order, update_max_debt_for_strategy ran before its add_strategy and reverted. The same bundle succeeds locally for all four calls.
Telegram "Call 0…3" vs report "Calls 1–4" Telegram used the 0-based on-chain index.

Changes

  • utils/proxy.py: minimal_proxy_implementation() decodes EIP-1167 clone bytecode.
  • utils/erc20_metadata.py: resolves clones before the selector check. It also reads name() in the same batch when the bytecode dispatches it, and retries without it on failure (bytes32 names such as MKR).
  • utils/llm/ai_explainer.py:
    • Labels lead with the on-chain name when the explorer label is only a contract type: wstETH/WETH Spark Looper (ysWETH, 18 dec) — LSTAaveLooper.
    • Param names are looked up by full signature.
    • Batch calls are never simulated one by one. A batch is simulated only as one ordered bundle from the executor. If the bundle request fails, no call is simulated: the call flow says Batch simulation: unavailable and the prompt tells the model not to infer an outcome. Calls after a bundle revert are marked not reached.
    • Two prompt rules: a unit stated by Protocol Context is verified and needs no hedging, and no "audited"/"low-fee"/"safe" claims unless a prompt section supports them.
  • utils/tenderly/simulation.py: logs Tenderly's error payload when a bundle returns no results.
  • utils/source_context.py:
    • Overloads are disambiguated by signature. A name with a single ABI entry behaves as before.
    • Vyper def extraction covers decorators, multi-line signatures with defaults, and docstrings. It normalizes CRLF (Etherscan serves the V3 vault source with CRLF) and prefers decorated definitions over interface stubs.
  • utils/llm/yearn_v3_context.py (new adapter, registered in protocol_context.py):
    • Selection is by call shape plus apiVersion() 3.x rather than protocol name, so it also covers third-party curators and Safe alerts.
    • It reads vault size, deposit limit, use_default_queue, and the default queue with debt/caps, plus each named strategy's registration, asset, size and whether it is itself an allocator vault.
    • It states every proposed value in asset units relative to that state.
  • protocols/timelock/timelock_alerts.py: batch calls are numbered from 1.
  • Docs: utils/llm/README.md (new 5f-2 section, source-context/metadata notes) and protocols/timelock/README.md.

Result on the same transaction (local end-to-end run)

Registers Flex WETH yVault on Yearn yETH Recovery Vault, appended to the default queue with 500 WETH max debt (20.4% of vault totalAssets). Registers wstETH/WETH Spark Looper on yvWETH-2 outside the default queue with 10,000 WETH max debt (8.6× vault totalAssets, equal to deposit limit). Neither allocation is immediate; debt moves only when a DEBT_MANAGER calls update_debt. … MEDIUM

The gist also gains a deterministic Protocol Context section with each vault's queue, debts and caps. The corrected report for the original alert is published as revision 2 of https://gist.wavey.info/E1U8blubtKZMJ82Px2vxfP3B.

Not addressed / worth knowing

  • Why the bundle request failed on the runner is still unknown. The runner syncs to main every 10 minutes, so it had the bundle code. The same bundle succeeds locally, and fetch_json already retries 429/5xx/timeouts. Check the runner logs around 2026-09-25 18:06 UTC for Bundle simulation unavailable, Tenderly bundle simulation returned, or an HTTP 4xx on simulate-bundle. With this PR, a repeat shows "unavailable" instead of false reverts.
  • The model still sometimes embellishes ("audited looper pattern", "institutional-facing vault") despite the new rule. Everything deterministic is now correct; this is prose-level, and the default model is a cheap one.
  • Call Flow amount hints (≈ 500 WETH) are not added for max_debt. The normalized values appear in the Protocol Context section instead.

Testing

  • uv run pytest tests/: 1432 passed. New tests cover clone resolution, name() read and fallback, EIP-1167 decoding, overload disambiguation, Vyper extraction (CRLF, multi-line signature, interface stub), the label format, bundle-only simulation (unavailable / not reached, no single-call fallback), Telegram numbering, and the adapter (proposal lines, rendering, protocol-agnostic grouping, non-V3/non-vault skip, read path).
  • uv run ruff format . and uv run ruff check . are clean. mypy is clean on the new module.
  • Live check against mainnet for this transaction: the adapter output, Vyper source context, and the full explain_batch_transaction run with the configured LLM.

🤖 Generated with Claude Code

spalen0 and others added 2 commits September 25, 2026 20:45
A Yearn timelock batch adding two strategies (add_strategy +
update_max_debt_for_strategy on two WETH vaults) produced a summary that
could not name either vault, called a Spark looper an Aave looper, hedged
that max_debt had "units unconfirmed", said the add_to_queue flag could
not be interpreted, and reported the debt-setting calls as unverified.

- Resolve EIP-1167 clones in ERC20 metadata. Every Yearn V3 vault is a
  clone, so vaults never got symbol/decimals and the model had nothing to
  size amounts with.
- Read name() with symbol/decimals and lead address labels with it when
  the explorer label is only the contract type ("Yearn V3 Vault",
  "LSTAaveLooper").
- Match ABI parameter names on the full signature when a function is
  overloaded. A Vyper default argument compiles to one entry per arity,
  and the first-entry lookup left add_to_queue unnamed.
- Extract Vyper function signatures and docstrings (CRLF-safe, preferring
  decorated definitions over interface stubs) for source context.
- Add a Yearn V3 protocol-context adapter keyed on call shape and
  apiVersion(). It states max_debt and related values in the vault asset
  against totalAssets, the default queue, peer caps and the deposit
  limit, and says what add_to_queue does given use_default_queue.
- When the bundle simulation is unavailable, name the earlier same-target
  call in an independent revert's diagnostic.
- Tell the model that Protocol Context units are verified, and forbid
  unsupported "audited"/"low-fee" claims.
- Number timelock batch calls from 1 in Telegram so they match the AI
  report's call flow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When the Tenderly bundle request failed, batch calls fell back to
independent simulations against current state. Out of batch order a
dependent call reverts falsely (update_max_debt_for_strategy before its
add_strategy), and the Yearn timelock alert reported exactly those false
reverts.

A batch is now only ever simulated as one ordered bundle from the
executor. If the bundle is unavailable, no call is simulated: the call
flow says so and the prompt tells the model not to infer an outcome.
Calls after a bundle revert are marked "not reached" rather than
re-simulated. Tenderly's error payload is logged when a bundle returns
no results, so the next fallback is diagnosable.

This replaces the same-target dependency hint added earlier in this
branch, which only existed to explain fallback reverts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@spalen0
spalen0 marked this pull request as ready for review September 25, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant