Repository navigation
app: reload proposer config file at runtime - #4831
KaloyanTanev wants to merge 3 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## kalo/revisioned-proposer-config #4831 +/- ##
===================================================================
+ Coverage 68.11% 68.23% +0.11%
===================================================================
Files 261 262 +1
Lines 34190 34310 +120
===================================================================
+ Hits 23289 23410 +121
+ Misses 10900 10899 -1
Partials 1 1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
722b960 to
88bc1b7
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues cause mixed-version configuration mismatches, can leave proposer settings stale or block recovery from invalid startup documents, and make notification tests flaky.
4 open issues
2 files not reviewed
app/lifecycle/orderstart_string.go: Generated fileapp/peerinfo/peerinfopb/v1/peerinfo.pb.go: Generated file
What changed in this PR
This PR connects the revisioned proposer config from #4829 to Charon’s runtime, allowing fee recipients, gas limits, and builder settings to update without restarting Charon.
Changes:
- Adds a file-watching service that reloads valid revisions and retains the current configuration when reloads fail.
- Converts core consumers to read runtime configuration through callbacks.
- Exchanges proposer revision hashes through peerinfo and updates metrics, documentation, and tests.
| File | Description |
|---|---|
docs/metrics.md |
Documents the proposer config mismatch metric. |
docs/configuration.md |
Documents runtime file reloads. |
dkg/dkg.go |
Supplies an empty config-hash callback for ceremonies. |
core/validatorapi/validatorapi.go |
Reads current gas limits and builder hosts. |
core/validatorapi/validatorapi_test.go |
Updates constructor calls for callbacks. |
core/validatorapi/validatorapi_internal_test.go |
Tests mismatch checks after configuration changes. |
core/tracker/tracker.go |
Reads builder hosts for each report. |
core/tracker/tracker_internal_test.go |
Tests reporting after builder changes. |
core/fetcher/fetcher.go |
Reads current builder settings and Gloas fee recipients. |
core/fetcher/fetcher_test.go |
Tests reloads and fork-specific fee recipient sources. |
core/fetcher/fetcher_internal_test.go |
Updates fetcher construction for callbacks. |
core/bcast/bcast.go |
Resolves builders from current configuration. |
core/bcast/bcast_test.go |
Tests broadcasts after builder changes. |
core/bcast/bcast_internal_test.go |
Updates builder configuration options. |
cmd/run.go |
Updates proposer config flag help. |
app/watchfile.go |
Adds a shared directory-based file watcher. |
app/watchfile_internal_test.go |
Tests notifications and watcher cancellation. |
app/proposerconfigservice.go |
Adds atomic snapshots, reload validation, and getters. |
app/proposerconfigservice_internal_test.go |
Tests reload acceptance, rejection, and getters. |
app/proposerconfig.go |
Constructs the runtime service during startup. |
app/proposerconfig_internal_test.go |
Adapts startup tests and adds parsing cases. |
app/peerinfo/peerinfopb/v1/peerinfo.proto |
Replaces the builder hash with a proposer hash. |
app/peerinfo/peerinfopb/v1/peerinfo.pb.go |
Updates generated protobuf bindings. |
app/peerinfo/peerinfo.go |
Exchanges and compares current revision hashes. |
app/peerinfo/peerinfo_test.go |
Updates peerinfo callback arguments. |
app/peerinfo/peerinfo_internal_test.go |
Tests revision mismatch metrics and logs. |
app/peerinfo/metrics.go |
Renames the configuration mismatch gauge. |
app/peerinfo/adhoc_test.go |
Updates peerinfo construction. |
app/metrics.go |
Refreshes builder gauges from runtime configuration. |
app/metrics_internal_test.go |
Retains URL credential-removal tests. |
app/lifecycle/orderstart_string.go |
Updates generated lifecycle names. |
app/lifecycle/order.go |
Adds the proposer config watcher lifecycle entry. |
app/builderregistration.go |
Reuses the shared file watcher. |
app/builderconfig.go |
Removes superseded flag-based builder helpers. |
app/builderconfig_internal_test.go |
Removes tests replaced by service and metrics tests. |
app/app.go |
Wires the service and callbacks into core components. |
Review effort: Balanced
88bc1b7 to
9e0820f
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Malformed proposer URLs can cause the metrics fallback to expose embedded passwords in Prometheus labels, so parse errors must use a credential-free placeholder or omit the metric.
1 open issue
4 resolved since last review
2 files not reviewed
app/lifecycle/orderstart_string.go: Generated fileapp/peerinfo/peerinfopb/v1/peerinfo.pb.go: Generated file
Review effort: Balanced
9e0820f to
bb0fc8f
Compare
|
OisinKyne
left a comment
There was a problem hiding this comment.
Points to raise (all low or design-level)
- Rollback protection only lasts until a restart. reload rejects a lower revision, but after a restart newProposerConfigService accepts whatever file is on disk. Restoring an older valid file is refused while charon runs and accepted after it restarts. That's probably acceptable, since the validator client follows the file anyway. Still, a one-line comment or doc note would stop anyone assuming it's a lasting high-water mark.
- Writes that aren't atomic cause brief error noise. watchFile reacts to each Write event, so a truncate-then-write can be read half-written. That logs an ERROR "Invalid reloaded proposer config" and sets proposer_config_invalid to 1, then the next coalesced event loads the full file and clears it. The final state is correct, but alerts on that gauge could fire falsely. A short debounce of a few hundred ms in watchFile would fix it. charon and the edit command write atomically, so this only affects hand edits.
- Some file swaps aren't detected. The watcher only reacts to Create or Write events on the file's own name. Kubernetes ConfigMap/Secret mounts update through a symlink swap on ..data, which never produces an event for that name. A host-side replace of a file mounted on its own in Docker isn't seen either. CDVN mounts the .charon directory, so it's fine. Worth a note if Helm ever mounts this file from a ConfigMap.
- Removed builders keep their metric. After a reload, app_tracker_builder_request_auths{builder=} stays in Prometheus at its last value until restart, because nothing deletes it. Not a problem, just a bit misleading on dashboards. Calling DeletePartialMatch on hosts that disappear would clean it up.
- Nit: the watcher goroutine is started in newProposerConfigService with the app-start context, not by the lifecycle, so its warnings are tagged with the app-start topic. Starting it inside Run would tie it to the lifecycle, though the watch would then have to be set up before the first load some other way.
Rollout concern (outside this PR)
In CDVN, validator clients render their settings from proposer-config.json once when they start (teku/run.sh waits for the file, then renders it). After a charon hot reload, charon is on the new revision and the validator client is still on the old one until it restarts. You'd see proposer-preferences fee-recipient and gas-limit mismatch warnings, and with Gloas the fetcher's verifyFeeRecipient checks against the new value. The release notes or edit proposer-config output should tell operators to restart their validator clients. The alternative is making the CDVN/LCDVN validator clients watch the file too. Otherwise "no restart needed" is only true for charon.





Makes
charon runserve the proposer config file instead of the seeding flags. Fee recipients, gas limit and builders come from the document, and a file watcher reloads it when a new valid revision lands, so changes apply without a restart. Core components read these values through funcs, and peerinfo exchanges the revision hash:app_peerinfo_builder_config_mismatchbecomesapp_peerinfo_proposer_config_mismatch.Stacked on #4829.
category: feature
ticket: #4324