Repository navigation
How to deal with the split-brain problem in kvrocks #329
Description
Hi guys,
while analyzing the use of a controller to manage KVrocks clusters in a multi-AZ deployment, we identified a risk of split-brain scenarios.
For example, in the diagram below, when a network partition occurs:
- The connection between the Load Balancer (LB) and Node N1 remains functional
- But the connection between the Controller Master and Node N1 fails due to network issues
This causes the Controller Master to mistakenly mark N1 as faulty and trigger failover, promoting N1's slave to master. However, N1 is actually healthy and continues to accept writes from the LB.
Result : Two active masters (split-brain) exist for the same shard, leading to data inconsistency.
In the current solution, this situation can occur in many scenarios, in addition to network partitions, there are also instances that hang for a while and then recover during failover and provide write services normally.
Activity
git-hulk commentedon Jul 31, 2025
Hi @Allen315, thanks for your report.
Yes, it is known issue. And perhaps we could improve this by using multiple nodes to detect the healthy status instead of the leader node only. Will do this when I get time and PR is also welcomed.
It's expected to promote a new master instance if it's unresponsive. You can also increase the failover detect times and interval to mitigate this issue.
Kaustubh1204 commentedon Feb 5, 2026
Hi @Allen315,
I’m going to work on this issue.
I’ll start by reviewing the current leader election and failure-handling logic to understand where split-brain can occur, and then propose a fix to strengthen leadership guarantees and prevent stale controllers from making cluster changes.
I’ll share my findings and a proposed approach here before opening a PR. Please let me know if there’s anything specific I should keep in mind.
Fix Split-Brain Vulnerability (Issue apache#329)
Kaustubh1204 commentedon Feb 11, 2026
Hi @Allen315, @git-hulk,
I’ve worked on fixing the split-brain issue (#329) and updated the workflow in .github/workflows/build.yml to safely handle fork PRs. The fix strengthens leadership guarantees and ensures that only the true leader can promote a master. The changes have been committed and referenced in PR #374.
CI Status:
Build: Passed
License check: Failed for controller/split_brain_test.go. This file is missing a valid license header and existed prior to my PR.
Lint/tests: Some lint issues may appear due to outdated .golangci.yml config in the repo; unrelated to my changes.
Next Steps:
Maintainers can fix the license header in split_brain_test.go.
My workflow and split-brain logic are safe to merge, as Go build passes and workflow syntax is valid.
Please review the PR when convenient. I’m happy to assist with license or lint updates if needed.
Thanks,
Kaustubh