Skip to content

fix(consensus): rebuild bootstrap snapshot when a vote races initial() - #2776

Open
wgr523 wants to merge 1 commit into
dev-upgradefrom
fix/bootstrap-snapshot-race-dev-upgrade
Open

wgr523 wants to merge 1 commit into
dev-upgradefrom
fix/bootstrap-snapshot-race-dev-upgrade

Conversation

@wgr523

@wgr523 wgr523 commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Proposed changes

On a fresh chain the v2 engine is initialised lazily by the first proposal (via verifyHeader), while the bft goroutine concurrently verifies votes for that block. VerifyVoteMessage takes no lock, so it can call getSnapshot before initial() has stored the V2 switch gap snapshot. The lookup misses, every vote for block 1 is dropped, no QC forms, and the chain stalls at height 1 permanently.

getSnapshot now rebuilds that one snapshot from the V2 switch block header (the same source initial() uses), stores it and returns it. Scope is narrow:

  • only the gap block initial() owns is rebuilt; every other missing snapshot still errors
  • an empty masternode set is refused rather than stored

Measured over 5 fresh 3-node bootstraps (private local net): unpatched stalled 1/5 (the run that logged a snapshot miss); patched passed 5/5, with one run logging the rebuild.

Types of changes

What types of changes does your code introduce to XDC network?
Put an ✅ in the boxes that apply

  • build: Changes that affect the build system or external dependencies
  • ci: Changes to CI configuration files and scripts
  • chore: Changes that don't change source code or tests
  • docs: Documentation only changes
  • feat: A new feature
  • fix: A bug fix
  • perf: A code change that improves performance
  • refactor: A code change that neither fixes a bug nor adds a feature
  • revert: Revert something
  • style: Changes that do not affect the meaning of the code
  • test: Adding missing tests or correcting existing tests

Impacted Components

Which parts of the codebase does this PR touch?
Put an ✅ in the boxes that apply

  • Consensus
  • Account
  • Network
  • Geth
  • Smart Contract
  • External components
  • Not sure (Please specify below)

Checklist

Put an ✅ in the boxes once you have confirmed below actions (or provide reasons on not doing so) that

  • This PR has sufficient test coverage (unit/integration test) OR I have provided reason in the PR description for not having test coverage
  • Tested on a private network from the genesis block and monitored the chain operating correctly for multiple epochs.
  • Provide an end-to-end test plan in the PR description on how to manually test it on the devnet/testnet.
  • Tested the backwards compatibility.
  • Tested with XDC nodes running this version co-exist with those running the previous version.
  • Relevant documentation has been updated as part of this PR
  • N/A

On a fresh chain nothing has initialised the v2 engine yet, so the first
proposal initialises it lazily through verifyHeader while the bft goroutine
concurrently verifies the votes for that same block. VerifyVoteMessage takes
no lock, so it can reach getSnapshot before initial() has called
StoreSnapshot for the V2 switch gap block. The lookup misses, every vote for
block 1 is dropped, no QC ever forms, and the chain stalls at height 1 with
an empty highestQuorumCert -- a state restarting cannot repair, because
block 1 is by then committed without a QC and is never re-proposed.

getSnapshot now rebuilds that one snapshot from the V2 switch block header,
the same source initial() reads, then stores and returns it. Scope is
narrow: only the gap block initial() owns is rebuilt, an empty masternode
set is refused rather than stored (an empty snapshot loads back fine and
would mask the hole permanently), and every other missing snapshot still
returns an error.

Measured over 5 fresh 3-node bootstraps: unpatched stalled 1/5, exactly the
run that logged a snapshot miss; patched passed 5/5, with one run logging
the rebuild -- the race fired and was absorbed without dropping a vote.
Reverting snapshot.go makes TestGetSnapshotRebuildsBootstrapSnapshot fail
with the production error, "getSnapshot() = not found".

Ported to dev-upgrade: when the rebuild does not apply, getSnapshot still
reports the miss as consensus.ErrGapSnapshotUnavailable. TestInitialFirstV2Block
asserted that no snapshot exists before initial(), which this change
deliberately reverses; it now expects the rebuilt gap 450 snapshot.

(cherry picked from commit fc838407115741a8616c2c3e97851d46abd22889)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 11, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 38150381-220d-4228-9702-a8c6656a8c99

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change affects concurrent consensus bootstrap behavior and warrants final human validation despite its focused tests.

0 open findings

What changed in this PR

Fixes a startup race that could prevent V2 consensus from forming the first quorum certificate.

Changes:

  • Rebuilds the missing bootstrap snapshot from the V2 switch header.
  • Rejects empty masternode snapshots.
  • Adds unit and integration coverage for pre-initialization lookup behavior.
File Description
consensus/​XDPoS/​engines/​engine_v2/​snapshot.go Adds bootstrap snapshot recovery.
consensus/​XDPoS/​engines/​engine_v2/​snapshot_test.go Tests recovery boundaries and validation.
consensus/​tests/​engine_v2_tests/​initial_test.go Verifies on-demand recovery before initialization.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants