Tracking: ev-metrics optimization roadmap (bugs, scalability, features, cleanup) #53

Tracking issue for the optimization review of ev-metrics: bugs, refactors and features, grouped by priority.

P0 = metrics are wrong, or the process crashes or hangs · P1 = scalability and reliability · P2 = features · P3 = refactor and hygiene
Effort: S < 1h · M ≈ half a day

Status (updated 2026-10-20)

• ✅ All roadmap issues are done: P0 (fix(celestia): send auth token on block timestamp lookups #55, fix: P0 correctness bugs (halt detection, hangs, panic, docs) #57), P1 (P1: verifier scalability, DA cache, client reuse, log format, security bumps #58), P2 (P2: self-observability, /health, receive delay, tx-block header checks, URL reda #59) and P3 (P3: remove debug env, cleanups, static non-root image, CI hardening #61). The ev-node v1.0.0 migration is fix: migrate to ev-node v1.0.0 DA client #56.
• ✅ fix(deps): upgrade ev-node to v1.0.0 and fix broken metrics #33 and chore(deps): bump github.com/ethereum/go-ethereum from 1.16.5 to 1.16.8 in the go_modules group across 1 directory #30 were closed as superseded.
• ⏳ Open PRs:
  • chore(deps): bump the go_modules group across 1 directory with 2 updates #60 (Dependabot: grpc 1.83.2, edwards25519 1.1.1) fixes the two open Dependabot alerts: gRPC DoS in xDS servers (high) and edwards25519 MultiScalarMult (low). Neither was flagged as reachable by govulncheck. Checks are green; ready to merge.
  • Ci/add to project #32 (add issues to the project board) still needs --project-id in place of --owner/--project-number; after it merges, add [P0] time_since_last_block never grows on chain halt when drift monitoring is enabled #34–P3: remove debug env, cleanups, static non-root image, CI hardening #61 to project 7 by hand.
• ⏳ Older issues: [FEATURE] add metric which exposes number of headers/data which did not validate successfully #5 looks covered by submission_failures_total (see the comment there, safe to close unless the metric name matters). [Feature Request]: Add E2E test to verify prometheus metrics are exposed correctly #23 (E2E test) is still open.
• This tracking issue can be closed once chore(deps): bump the go_modules group across 1 directory with 2 updates #60 and Ci/add to project #32 are dealt with.

Upgrade notes from this roadmap

• Go 1.26 is required (P1: verifier scalability, DA cache, client reuse, log format, security bumps #58).
• Dashboards using the submission_duration_seconds summary now see NaN quantiles when no submission was verified in the last 60s. Use the new last_submission_duration_seconds gauge for the latest value ([P2] Decide: RefreshSubmissionDuration fabricates summary samples #48).
• Header submission counts rise (submission_*{type="header"}, unsubmitted_*{blob_type="header"}) because blocks with transactions now verify their header blob too ([P2] Verify header blob for blocks with transactions too #47).
• Endpoint labels show only scheme://host[:port]; alerts that matched a full URL with a path need updating ([P2] Strip credentials/paths from endpoint URLs used as metric labels and in logs #49).
• New: --log-format=json|console ([P1] Reduce per-block log volume and add --log-format=json #43), --health-max-header-age ([P2] Make /health reflect subscription and header freshness #46), /health now reflects subscription state (use it as a readiness probe, not liveness), and the self-observability metrics from [P2] Add self-observability metrics (queue length, in-flight, WS reconnects, last header) #45.
• Per-block and per-poll success logs are now Debug level; use --verbose to see them ([P1] Reduce per-block log volume and add --log-format=json #43).
• EVMETRICS_DEBUG no longer does anything; use --verbose ([P3] Remove EVMETRICS_DEBUG / log.Printf from pkg/metrics in favour of zerolog #50).
• The container runs as UID 65532 on a distroless static image (was root on alpine). Deployments that mount root-owned files or volumes need adjusting ([P3] Dockerfile: static non-root build; CI: run tests with -race #52).
• The JSON-RPC health check is stricter: a 200 response with a JSON-RPC error body, or a body that is not JSON, now counts as unavailable ([P3] Small cleanups in metrics.go, evnode and evm clients #51).
• CI: make test runs with -race, golangci-lint is pinned to v2.15.0, actions moved to Node 24 versions, and a govulncheck workflow fails on any reachable vulnerability except GO-2026-5932 (x/crypto, no fix yet). Delete that allowance once a fix ships ([P3] Dockerfile: static non-root build; CI: run tests with -race #52).

P0 — Correctness ✅ done

• [P0] time_since_last_block never grows on chain halt when drift monitoring is enabled #34 time_since_last_block never grows on chain halt when drift monitoring is enabled (S), fixed by fix: P0 correctness bugs (halt detection, hangs, panic, docs) #57
• [P0] Celestia GetBlockTimestamp ignores --celestia-token (all verifications fail with auth enabled) #35 Celestia GetBlockTimestamp ignores --celestia-token (S), fixed by fix(celestia): send auth token on block timestamp lookups #55
• [P0] Exporter failure leaves process hung; --duration exits non-zero #36 Exporter failure leaves process hung; --duration exits non-zero (S), fixed by fix: P0 correctness bugs (halt detection, hangs, panic, docs) #57
• [P0] Verifier re-queue goroutine can panic (send on closed channel) during shutdown #37 Verifier re-queue can panic on shutdown (S, can be folded into [P1] Verifier: decouple retry waits from workers (throughput cap + skewed block_time) #39), fixed by fix: P0 correctness bugs (halt detection, hangs, panic, docs) #57
• [P0] README documents non-existent flags (--enable-metrics, --debug), modes and metrics #38 README documents non-existent flags, modes and metrics (S), fixed by fix: P0 correctness bugs (halt detection, hangs, panic, docs) #57

P1 — Scalability and reliability ✅ done

• [P1] Verifier: decouple retry waits from workers (throughput cap + skewed block_time) #39 Decouple retry waits from verifier workers (M). Includes small interfaces so the verifier can be tested. Fixed by P1: verifier scalability, DA cache, client reuse, log format, security bumps #58
• [P1] Verifier: backfill blocks missed during WebSocket reconnect #41 Backfill blocks missed during WebSocket reconnect (S–M, after [P1] Verifier: decouple retry waits from workers (throughput cap + skewed block_time) #39), fixed by P1: verifier scalability, DA cache, client reuse, log format, security bumps #58
• [P1] Verifier: remove duplicate ev-node call and cache Celestia lookups per DA height #40 Remove duplicate ev-node call and cache Celestia lookups per DA height (M), fixed by P1: verifier scalability, DA cache, client reuse, log format, security bumps #58
• [P1] Drift/balance: reuse clients, add per-call timeouts, poll concurrently, support TLS gRPC #42 Drift and balance: reuse clients, per-call timeouts, concurrent polling, TLS gRPC (S–M), fixed by P1: verifier scalability, DA cache, client reuse, log format, security bumps #58
• [P1] Reduce per-block log volume and add --log-format=json #43 Reduce per-block log volume and add --log-format=json (S), fixed by P1: verifier scalability, DA cache, client reuse, log format, security bumps #58
• [P1] Bump vulnerable deps: go-ethereum ≥1.17.0, grpc ≥1.83.1, x/net ≥0.60.0 (supersedes #30) #54 Bump vulnerable dependencies: go-ethereum ≥1.17.0, grpc, x/net, cometbft (S; supersedes chore(deps): bump github.com/ethereum/go-ethereum from 1.16.5 to 1.16.8 in the go_modules group across 1 directory #30). Fixed by P1: verifier scalability, DA cache, client reuse, log format, security bumps #58

P2 — Features ✅ done

• [P2] Emit declared-but-unused metrics: block_receive_delay and SLO threshold gauges #44 Emit declared-but-unused metrics (block_receive_delay and SLO gauges) (S, resolves part of [P0] README documents non-existent flags (--enable-metrics, --debug), modes and metrics #38). Fixed by P2: self-observability, /health, receive delay, tx-block header checks, URL reda #59
• [P2] Add self-observability metrics (queue length, in-flight, WS reconnects, last header) #45 Self-observability metrics: queue length, in-flight blocks, WS reconnects, last header (S). Fixed by P2: self-observability, /health, receive delay, tx-block header checks, URL reda #59
• [P2] Make /health reflect subscription and header freshness #46 Make /health reflect subscription and header freshness (S). Fixed by P2: self-observability, /health, receive delay, tx-block header checks, URL reda #59
• [P2] Verify header blob for blocks with transactions too #47 Verify the header blob for blocks with transactions (S). Fixed by P2: self-observability, /health, receive delay, tx-block header checks, URL reda #59
• [P2] Decide: RefreshSubmissionDuration fabricates summary samples #48 RefreshSubmissionDuration fabricated summary samples; replaced by a last_submission_duration_seconds gauge (S). Fixed by P2: self-observability, /health, receive delay, tx-block header checks, URL reda #59
• [P2] Strip credentials/paths from endpoint URLs used as metric labels and in logs #49 Strip credentials and paths from endpoint URLs used as labels and in logs (S). Fixed by P2: self-observability, /health, receive delay, tx-block header checks, URL reda #59

P3 — Refactor and hygiene ✅ done

• [P3] Remove EVMETRICS_DEBUG / log.Printf from pkg/metrics in favour of zerolog #50 Remove EVMETRICS_DEBUG / log.Printf in favour of zerolog (S). Fixed by P3: remove debug env, cleanups, static non-root image, CI hardening #61
• [P3] Small cleanups in metrics.go, evnode and evm clients #51 Small cleanups in metrics.go and the evnode and evm clients (S). Fixed by P3: remove debug env, cleanups, static non-root image, CI hardening #61
• [P3] Dockerfile: static non-root build; CI: run tests with -race #52 Static non-root Docker build; CI: -race, pinned golangci-lint, govulncheck, Node 24 actions (S). Fixed by P3: remove debug env, cleanups, static non-root image, CI hardening #61

Related existing issues

• [Feature Request]: Add E2E test to verify prometheus metrics are exposed correctly #23 E2E test for Prometheus metrics exposure
• [FEATURE] add metric which exposes number of headers/data which did not validate successfully #5 Metric for headers and data that failed validation

Order followed

1. P0 ✅ (fix(celestia): send auth token on block timestamp lookups #55, fix: migrate to ev-node v1.0.0 DA client #56, fix: P0 correctness bugs (halt detection, hangs, panic, docs) #57)
2. P1 ✅ (P1: verifier scalability, DA cache, client reuse, log format, security bumps #58)
3. P2 ✅ (P2: self-observability, /health, receive delay, tx-block header checks, URL reda #59)
4. P3 ✅ (P3: remove debug env, cleanups, static non-root image, CI hardening #61)
5. Remaining housekeeping: merge chore(deps): bump the go_modules group across 1 directory with 2 updates #60, fix and merge Ci/add to project #32

Intentionally out of scope (YAGNI for now)

• Prometheus native histograms instead of the histogram + summary pairs
• A config file, or time.Duration-typed flags (breaking change)
• Configurable SLO thresholds (wait until a second chain needs different ones)

Definition of done (per PR)

• make lint-fix is clean and go test -race ./... passes.
• New or changed flags and metrics are documented in README.md.
• Every behaviour change has a test.