[auricom]

Implements the P2 issues from the optimization roadmap (#53).

Changes

Receive delay and SLO metrics (#44)

• ev_metrics_block_receive_delay_seconds is now recorded for every header received from the subscription. Negative values from clock skew are recorded as 0, and backfilled headers are not counted.
• block_time_slo_seconds and block_receive_delay_slo_seconds are set at verifier start.
• header.Time has 1-second resolution, so the delay is only accurate to about ±1s (noted in the README). The "not yet emitted" notes are gone.

Self-observability metrics and a real /health (#45, #46)

• New metrics, all visible from startup:
  • ev_metrics_verifier_queue_length (sampled every 10s)
  • ev_metrics_verifier_inflight_blocks
  • ev_metrics_ws_reconnects_total
  • ev_metrics_last_header_received_timestamp
  • ev_metrics_verifier_backfilled_blocks_total
  • ev_metrics_verifier_backfill_dropped_total
• /health returns 503 until the first subscription is up, and while the subscription is down. Otherwise it returns 200.
• New flag --health-max-header-age (seconds, default 0). 0 disables the age check, so only the subscription state matters. When set, /health also returns 503 if no header has arrived within that time. The check counts from the later of the first subscription and the last header.
• Use /health as a readiness probe, not liveness. With the age check on, a halted chain fails it, and restarting the monitor would not help. The README says so.
• StartServer gains a healthMaxHeaderAge parameter. Nothing outside this repo calls it.

Header verification for blocks with transactions (#47)

• Blocks with transactions now verify both their header blob and their data blob. Each outcome is recorded under its own type label. Blocks without transactions still verify only the header.
• Progress is tracked per job (headerDone, dataDone), so a retry only redoes the pending part and nothing is counted twice. When retries run out, only the still-unverified types are recorded as missing.
• The cost is one extra Celestia request per transaction block. The Celestia cache from P1: verifier scalability, DA cache, client reuse, log format, security bumps #58 makes the DA timestamp lookup free, but the header check is not cached.

Endpoint credential redaction (#49)

• Endpoint URLs in metric labels and logs now show only scheme://host[:port]. Userinfo, path, query and fragment are dropped, so API keys in URLs are no longer exposed. This covers the drift and balance labels and logs, and the balance startup log.
• Errors are redacted too, because Go's HTTP errors embed the full URL (RedactError keeps errors.Is and errors.As working).
• If two endpoints reduce to the same label, the later ones get #<index> so they stay distinct. Nothing is hashed.
• The cosmos client's gRPC target also drops userinfo and query, so its debug logs and connect errors can't leak them.

Submission duration is no longer fabricated (#48)

• Removed RefreshSubmissionDuration, which re-recorded the last duration every 10s. That biased the summary quantiles toward the last value and inflated _count and _sum.
• ev_metrics_submission_duration_seconds now contains only real observations.
• New gauge ev_metrics_last_submission_duration_seconds{chain_id,type} holds the latest value. It is absent until the first submission is verified.

Behaviour changes to watch for

• [P2] Decide: RefreshSubmissionDuration fabricates summary samples #48 breaks dashboards that use the summary. Its quantiles are NaN when no submission was verified in the last 60 seconds, instead of repeating the last value. Switch those panels to last_submission_duration_seconds for the latest value.
• [P2] Verify header blob for blocks with transactions too #47 raises header submission counts. submission_*{type="header"}, the header DA height and duration, and unsubmitted_*{blob_type="header"} rise after upgrading because transaction blocks now count there too.
• Endpoint label values change ([P2] Strip credentials/paths from endpoint URLs used as metric labels and in logs #49). Any alert or dashboard that matches the full endpoint URL, including its path, needs updating. Labels that were already scheme://host[:port] are unchanged.

Tests

• New tests:
  • Receive delay and the SLO gauges.
  • Each new metric through reg.Gather().
  • /health: 503 before the subscription, 200 after, 503 when down, and the age check on and off.
  • A reconnect increments the counter and restores health.
  • Transaction blocks: header verified before data and counted once, header missing while data is fine, both succeeding, and a non-transaction block.
  • Redaction: table tests, collision handling, errors.Is and errors.As, and checks that no label or log line contains a secret placed in the URL.
  • The summary holds only real observations, and the last-duration gauge holds the latest value.
• Results: go build, go vet, go test -race ./... (8 packages), golangci-lint (0 issues), gofmt and go mod tidy are clean.
• Known limitation: go test -count=3 on pkg/metrics still panics on a duplicate registration. TestStartServer_ExporterErrorStopsServer calls StartServer, which registers on the global Prometheus registry, so it can only run once per process. This was already the case before this PR, and CI runs the tests once.

Fixes #44
Fixes #45
Fixes #46
Fixes #47
Fixes #48
Fixes #49

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 1e102185-2a16-4a21-b1fe-0778cf7325bf

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}