Skip to content

feat: stamp node_version and node_commit on every json log line - #225

Open
jacderida wants to merge 1 commit into
WithAutonomi:mainfrom
jacderida:v2-1153
Open

jacderida wants to merge 1 commit into
WithAutonomi:mainfrom
jacderida:v2-1153

Conversation

@jacderida

Copy link
Copy Markdown
Member

Linear issue

Closes V2-1153

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Proposed T1: the only behavioural change is two extra fields on each JSON log line. No wire, storage, payment, routing or upgrade-mechanism change.

Compatibility

  • Wire: none
  • Storage: none
  • API: none. JSON log lines gain node_version and node_commit fields; the text format and the startup line's existing version / commit fields (parsed by the ant-client beta forwarder) are unchanged.

Semver impact

  • breaking
  • feature
  • fix

Summary

Every document in production Elasticsearch carries tag.node_version = "unknown", so the build a node is running cannot be read from telemetry. That made the 0.18.1 staged rollout unreadable: fleet-wide relay recv stream closed errors rose 6x across the window, which looked like a regression in the build that fixed them; only splitting by the build each service was actually running (inferred from process start time) showed the new build was ~4x better (V2-986).

The tag is a static Telegraf [global_tags] value set at provisioning time in saorsa-testnet-registry and never passed by any caller. Even if it were passed, a provisioning-time string can never be right across an auto-upgrade, so the process itself has to report its build.

  • ant_node::logging::WithBuildInfo wraps the JSON event formatter: it renders the inner event into a thread-local buffer and replaces the closing brace with a compile-time ,"node_version":"<CARGO_PKG_VERSION>","node_commit":"<ANT_GIT_COMMIT>"} tail. One memcpy per line on top of the serialisation already done; re-entrancy falls back to a fresh String rather than panicking; non-object output passes through untouched.
  • Why a wrapper: tracing-subscriber's JSON formatter has no hook for constant fields, and span fields do not survive tokio::spawn, so a root span would miss most lines.
  • Both JSON sinks (stdout and daily-rolling file) use it via json_event_format() in main.rs.
  • Auto-upgrade: the upgrade path either execs the new binary or exits for the service manager to restart it, so the new process stamps its own constants — correct on both sides of the restart by construction.

Companion change: jacderida/saorsa-testnet-registry adds node_version, node_commit to Telegraf tag_keys so the fields land as tag.node_version.keyword / tag.node_commit.keyword and beat the static global tag. Until that config is re-provisioned on ant-prod-01, the values still land as top-level node_version / node_commit fields, so they are queryable either way. The bootstrap-upgrade-check / track-upgrades-setup skill notes saying tag.node_version is static remain true until both have rolled out.

Test evidence

  • 3 new unit tests in src/logging.rs: every line is valid JSON and keeps message/level/target/custom fields; the stamp lands after a nested "span":{...} object; node_version equals CARGO_PKG_VERSION and parses as semver.
  • cargo test: 1088 lib + 16 integration passed, 0 failed.
  • Smoke run in development mode: stdout sink 295/295 lines and file sink 81/81 lines carried node_version=0.18.1 node_commit=31fcbae, 0 invalid JSON (jq-checked); text format byte-identical.
  • cargo clippy --all-features -- -D clippy::panic -D clippy::unwrap_used -D clippy::expect_used clean; CI's cargo clippy --all-targets --all-features -- -D warnings clean; cargo build --no-default-features compiles.

Sample line:

{"timestamp":"2026-09-11T23:58:07.002929Z","level":"INFO","message":"Bootstrap peers provided via CLI","count":1,"target":"ant_node","node_version":"0.18.1","node_commit":"31fcbae"}

New dependency

none

ADR

n/a

Mitigation / rollback

Revert this commit; the two fields disappear and Telegraf's static global tag takes over again. No stored data or peers are affected.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MYvuVpFYXXNrY9YFVBswvp

Every document in production Elasticsearch carries `tag.node_version = "unknown"`, so the build a
node is running cannot be read from telemetry at all. That made the 0.18.1 staged rollout
unreadable: fleet-wide `relay recv stream closed` errors rose 6x across the window, which looked
like a regression in the build that fixed them, and only splitting by the build each service was
actually running (inferred from process start time) showed the new build was ~4x better.

The tag is a static Telegraf `[global_tags]` value set at provisioning time and never passed by any
caller. Even if it were passed, a provisioning-time string can never be right across an
auto-upgrade, so the process itself has to report its build.

`WithBuildInfo` wraps the JSON event formatter: it renders the inner event into a thread-local
buffer and replaces the closing brace with a compile-time
`,"node_version":"<CARGO_PKG_VERSION>","node_commit":"<ANT_GIT_COMMIT>"}` tail. Cost is one memcpy
per line on top of the serialisation already done. tracing-subscriber's JSON formatter has no hook
for constant fields and span fields do not survive `tokio::spawn`, hence the wrapper. Both JSON
sinks (stdout and rolling file) use it; the text format and the startup line's `version`/`commit`
fields are unchanged, so the beta forwarder in ant-client keeps working. After an auto-upgrade the
new process stamps its own constants, so the value is correct on both sides of the restart by
construction.

Telegraf promotes the fields to tags via `tag_keys` in saorsa-testnet-registry; until that config is
re-provisioned the values still land as top-level `node_version` / `node_commit` fields.

Test evidence:
- 3 new unit tests (fields kept, stamp after a nested `span` object, version is semver)
- cargo test: 1088 lib + 16 integration passed, 0 failed
- smoke run in development mode: 295/295 stdout lines and 81/81 file-sink lines carried
  `node_version=0.18.1 node_commit=31fcbae`, 0 invalid JSON; text format byte-identical
- clippy clean under both the CLAUDE.md deny set and CI's `--all-targets -D warnings`;
  `--no-default-features` still builds

Closes V2-1153

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MYvuVpFYXXNrY9YFVBswvp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant