Conversation
Every document in production Elasticsearch carries `tag.node_version = "unknown"`, so the build a node is running cannot be read from telemetry at all. That made the 0.18.1 staged rollout unreadable: fleet-wide `relay recv stream closed` errors rose 6x across the window, which looked like a regression in the build that fixed them, and only splitting by the build each service was actually running (inferred from process start time) showed the new build was ~4x better. The tag is a static Telegraf `[global_tags]` value set at provisioning time and never passed by any caller. Even if it were passed, a provisioning-time string can never be right across an auto-upgrade, so the process itself has to report its build. `WithBuildInfo` wraps the JSON event formatter: it renders the inner event into a thread-local buffer and replaces the closing brace with a compile-time `,"node_version":"<CARGO_PKG_VERSION>","node_commit":"<ANT_GIT_COMMIT>"}` tail. Cost is one memcpy per line on top of the serialisation already done. tracing-subscriber's JSON formatter has no hook for constant fields and span fields do not survive `tokio::spawn`, hence the wrapper. Both JSON sinks (stdout and rolling file) use it; the text format and the startup line's `version`/`commit` fields are unchanged, so the beta forwarder in ant-client keeps working. After an auto-upgrade the new process stamps its own constants, so the value is correct on both sides of the restart by construction. Telegraf promotes the fields to tags via `tag_keys` in saorsa-testnet-registry; until that config is re-provisioned the values still land as top-level `node_version` / `node_commit` fields. Test evidence: - 3 new unit tests (fields kept, stamp after a nested `span` object, version is semver) - cargo test: 1088 lib + 16 integration passed, 0 failed - smoke run in development mode: 295/295 stdout lines and 81/81 file-sink lines carried `node_version=0.18.1 node_commit=31fcbae`, 0 invalid JSON; text format byte-identical - clippy clean under both the CLAUDE.md deny set and CI's `--all-targets -D warnings`; `--no-default-features` still builds Closes V2-1153 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MYvuVpFYXXNrY9YFVBswvp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear issue
Closes V2-1153
Risk tier
Proposed T1: the only behavioural change is two extra fields on each JSON log line. No wire, storage, payment, routing or upgrade-mechanism change.
Compatibility
node_versionandnode_commitfields; the text format and the startup line's existingversion/commitfields (parsed by the ant-client beta forwarder) are unchanged.Semver impact
Summary
Every document in production Elasticsearch carries
tag.node_version = "unknown", so the build a node is running cannot be read from telemetry. That made the 0.18.1 staged rollout unreadable: fleet-widerelay recv stream closederrors rose 6x across the window, which looked like a regression in the build that fixed them; only splitting by the build each service was actually running (inferred from process start time) showed the new build was ~4x better (V2-986).The tag is a static Telegraf
[global_tags]value set at provisioning time insaorsa-testnet-registryand never passed by any caller. Even if it were passed, a provisioning-time string can never be right across an auto-upgrade, so the process itself has to report its build.ant_node::logging::WithBuildInfowraps the JSON event formatter: it renders the inner event into a thread-local buffer and replaces the closing brace with a compile-time,"node_version":"<CARGO_PKG_VERSION>","node_commit":"<ANT_GIT_COMMIT>"}tail. One memcpy per line on top of the serialisation already done; re-entrancy falls back to a freshStringrather than panicking; non-object output passes through untouched.tokio::spawn, so a root span would miss most lines.json_event_format()inmain.rs.execs the new binary or exits for the service manager to restart it, so the new process stamps its own constants — correct on both sides of the restart by construction.Companion change: jacderida/saorsa-testnet-registry adds
node_version,node_committo Telegraftag_keysso the fields land astag.node_version.keyword/tag.node_commit.keywordand beat the static global tag. Until that config is re-provisioned on ant-prod-01, the values still land as top-levelnode_version/node_commitfields, so they are queryable either way. Thebootstrap-upgrade-check/track-upgrades-setupskill notes sayingtag.node_versionis static remain true until both have rolled out.Test evidence
src/logging.rs: every line is valid JSON and keepsmessage/level/target/custom fields; the stamp lands after a nested"span":{...}object;node_versionequalsCARGO_PKG_VERSIONand parses as semver.cargo test: 1088 lib + 16 integration passed, 0 failed.node_version=0.18.1 node_commit=31fcbae, 0 invalid JSON (jq-checked); text format byte-identical.cargo clippy --all-features -- -D clippy::panic -D clippy::unwrap_used -D clippy::expect_usedclean; CI'scargo clippy --all-targets --all-features -- -D warningsclean;cargo build --no-default-featurescompiles.Sample line:
{"timestamp":"2026-09-11T23:58:07.002929Z","level":"INFO","message":"Bootstrap peers provided via CLI","count":1,"target":"ant_node","node_version":"0.18.1","node_commit":"31fcbae"}New dependency
none
ADR
n/a
Mitigation / rollback
Revert this commit; the two fields disappear and Telegraf's static global tag takes over again. No stored data or peers are affected.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MYvuVpFYXXNrY9YFVBswvp