Skip to content

app: reload proposer config file at runtime - #4831

Open
KaloyanTanev wants to merge 3 commits into
kalo/revisioned-proposer-configfrom
kalo/proposer-config-runtime
Open

KaloyanTanev wants to merge 3 commits into
kalo/revisioned-proposer-configfrom
kalo/proposer-config-runtime

Conversation

@KaloyanTanev

Copy link
Copy Markdown
Collaborator

Makes charon run serve the proposer config file instead of the seeding flags. Fee recipients, gas limit and builders come from the document, and a file watcher reloads it when a new valid revision lands, so changes apply without a restart. Core components read these values through funcs, and peerinfo exchanges the revision hash: app_peerinfo_builder_config_mismatch becomes app_peerinfo_proposer_config_mismatch.

Stacked on #4829.

category: feature
ticket: #4324

@KaloyanTanev KaloyanTanev self-assigned this Nov 10, 2026
@github-actions github-actions Bot added the branch-invalid PR raised against invalid branch. Not a main or release branch. label Nov 10, 2026
@KaloyanTanev
KaloyanTanev added this pull request to stack #4832 November 10, 2026 10:03
@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.92617% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.23%. Comparing base (b257d42) to head (bb0fc8f).

Files with missing lines Patch % Lines
app/app.go 0.00% 11 Missing ⚠️
app/proposerconfigservice.go 90.21% 9 Missing ⚠️
app/watchfile.go 82.75% 5 Missing ⚠️
app/proposerconfig.go 92.30% 3 Missing ⚠️
app/peerinfo/peerinfopb/v1/peerinfo.pb.go 0.00% 1 Missing ⚠️
core/bcast/bcast.go 83.33% 1 Missing ⚠️
core/fetcher/fetcher.go 96.29% 1 Missing ⚠️
core/tracker/tracker.go 87.50% 1 Missing ⚠️
core/validatorapi/validatorapi.go 90.90% 1 Missing ⚠️
Additional details and impacted files
@@                         Coverage Diff                         @@
##           kalo/revisioned-proposer-config    #4831      +/-   ##
===================================================================
+ Coverage                            68.11%   68.23%   +0.11%     
===================================================================
  Files                                  261      262       +1     
  Lines                                34190    34310     +120     
===================================================================
+ Hits                                 23289    23410     +121     
+ Misses                               10900    10899       -1     
  Partials                                 1        1              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved issues cause mixed-version configuration mismatches, can leave proposer settings stale or block recovery from invalid startup documents, and make notification tests flaky.

4 open issues
2 files not reviewed
  • app/lifecycle/orderstart_string.go: Generated file
  • app/peerinfo/peerinfopb/v1/peerinfo.pb.go: Generated file
What changed in this PR

This PR connects the revisioned proposer config from #4829 to Charon’s runtime, allowing fee recipients, gas limits, and builder settings to update without restarting Charon.

Changes:

  • Adds a file-watching service that reloads valid revisions and retains the current configuration when reloads fail.
  • Converts core consumers to read runtime configuration through callbacks.
  • Exchanges proposer revision hashes through peerinfo and updates metrics, documentation, and tests.
File Description
docs/​metrics.md Documents the proposer config mismatch metric.
docs/​configuration.md Documents runtime file reloads.
dkg/​dkg.go Supplies an empty config-hash callback for ceremonies.
core/​validatorapi/​validatorapi.go Reads current gas limits and builder hosts.
core/​validatorapi/​validatorapi_test.go Updates constructor calls for callbacks.
core/​validatorapi/​validatorapi_internal_test.go Tests mismatch checks after configuration changes.
core/​tracker/​tracker.go Reads builder hosts for each report.
core/​tracker/​tracker_internal_test.go Tests reporting after builder changes.
core/​fetcher/​fetcher.go Reads current builder settings and Gloas fee recipients.
core/​fetcher/​fetcher_test.go Tests reloads and fork-specific fee recipient sources.
core/​fetcher/​fetcher_internal_test.go Updates fetcher construction for callbacks.
core/​bcast/​bcast.go Resolves builders from current configuration.
core/​bcast/​bcast_test.go Tests broadcasts after builder changes.
core/​bcast/​bcast_internal_test.go Updates builder configuration options.
cmd/​run.go Updates proposer config flag help.
app/​watchfile.go Adds a shared directory-based file watcher.
app/​watchfile_internal_test.go Tests notifications and watcher cancellation.
app/​proposerconfigservice.go Adds atomic snapshots, reload validation, and getters.
app/​proposerconfigservice_internal_test.go Tests reload acceptance, rejection, and getters.
app/​proposerconfig.go Constructs the runtime service during startup.
app/​proposerconfig_internal_test.go Adapts startup tests and adds parsing cases.
app/​peerinfo/​peerinfopb/​v1/​peerinfo.proto Replaces the builder hash with a proposer hash.
app/​peerinfo/​peerinfopb/​v1/​peerinfo.pb.go Updates generated protobuf bindings.
app/​peerinfo/​peerinfo.go Exchanges and compares current revision hashes.
app/​peerinfo/​peerinfo_test.go Updates peerinfo callback arguments.
app/​peerinfo/​peerinfo_internal_test.go Tests revision mismatch metrics and logs.
app/​peerinfo/​metrics.go Renames the configuration mismatch gauge.
app/​peerinfo/​adhoc_test.go Updates peerinfo construction.
app/​metrics.go Refreshes builder gauges from runtime configuration.
app/​metrics_internal_test.go Retains URL credential-removal tests.
app/​lifecycle/​orderstart_string.go Updates generated lifecycle names.
app/​lifecycle/​order.go Adds the proposer config watcher lifecycle entry.
app/​builderregistration.go Reuses the shared file watcher.
app/​builderconfig.go Removes superseded flag-based builder helpers.
app/​builderconfig_internal_test.go Removes tests replaced by service and metrics tests.
app/​app.go Wires the service and callbacks into core components.

Review effort: Balanced

Comment thread app/proposerconfig.go Resolved
Comment thread app/proposerconfigservice.go Resolved
Comment thread app/watchfile_internal_test.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Malformed proposer URLs can cause the metrics fallback to expose embedded passwords in Prometheus labels, so parse errors must use a credential-free placeholder or omit the metric.

1 open issue
4 resolved since last review
2 files not reviewed
  • app/lifecycle/orderstart_string.go: Generated file
  • app/peerinfo/peerinfopb/v1/peerinfo.pb.go: Generated file

Review effort: Balanced

Comment thread app/metrics.go Resolved
@KaloyanTanev
KaloyanTanev force-pushed the kalo/proposer-config-runtime branch from 9e0820f to bb0fc8f Compare November 10, 2026 11:30
@KaloyanTanev
KaloyanTanev marked this pull request as ready for review November 10, 2026 11:30
@sonarqubecloud

Copy link
Copy Markdown

@OisinKyne OisinKyne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Points to raise (all low or design-level)

  1. Rollback protection only lasts until a restart. reload rejects a lower revision, but after a restart newProposerConfigService accepts whatever file is on disk. Restoring an older valid file is refused while charon runs and accepted after it restarts. That's probably acceptable, since the validator client follows the file anyway. Still, a one-line comment or doc note would stop anyone assuming it's a lasting high-water mark.
  2. Writes that aren't atomic cause brief error noise. watchFile reacts to each Write event, so a truncate-then-write can be read half-written. That logs an ERROR "Invalid reloaded proposer config" and sets proposer_config_invalid to 1, then the next coalesced event loads the full file and clears it. The final state is correct, but alerts on that gauge could fire falsely. A short debounce of a few hundred ms in watchFile would fix it. charon and the edit command write atomically, so this only affects hand edits.
  3. Some file swaps aren't detected. The watcher only reacts to Create or Write events on the file's own name. Kubernetes ConfigMap/Secret mounts update through a symlink swap on ..data, which never produces an event for that name. A host-side replace of a file mounted on its own in Docker isn't seen either. CDVN mounts the .charon directory, so it's fine. Worth a note if Helm ever mounts this file from a ConfigMap.
  4. Removed builders keep their metric. After a reload, app_tracker_builder_request_auths{builder=} stays in Prometheus at its last value until restart, because nothing deletes it. Not a problem, just a bit misleading on dashboards. Calling DeletePartialMatch on hosts that disappear would clean it up.
  5. Nit: the watcher goroutine is started in newProposerConfigService with the app-start context, not by the lifecycle, so its warnings are tagged with the app-start topic. Starting it inside Run would tie it to the lifecycle, though the watch would then have to be set up before the first load some other way.

Rollout concern (outside this PR)

In CDVN, validator clients render their settings from proposer-config.json once when they start (teku/run.sh waits for the file, then renders it). After a charon hot reload, charon is on the new revision and the validator client is still on the old one until it restarts. You'd see proposer-preferences fee-recipient and gas-limit mismatch warnings, and with Gloas the fetcher's verifyFeeRecipient checks against the new value. The release notes or edit proposer-config output should tell operators to restart their validator clients. The alternative is making the CDVN/LCDVN validator clients watch the file too. Otherwise "no restart needed" is only true for charon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

branch-invalid PR raised against invalid branch. Not a main or release branch.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants