What Is the Split-Brain Problem in Distributed Databases?
Understand the split-brain problem, why network partitions cause it, and how quorum and fencing prevent conflicting writes.
Expected Interview Answer
Split-brain occurs when a network partition or failed failover causes two nodes in a distributed database cluster to each believe they are the sole primary and independently accept writes, producing two diverging, conflicting copies of the data.
This typically happens when a network partition isolates the primary from the rest of the cluster; the remaining nodes, unable to reach the primary, elect a new one, but the original primary is still alive and still accepting client writes because it does not know it has been replaced. Once the partition heals, the cluster has two histories of writes that must somehow be reconciled, and some writes are often unrecoverably lost or must be manually merged. Distributed systems avoid this using quorum-based consensus (requiring a majority of nodes to agree before promoting a new primary) and fencing mechanisms that forcibly cut off the old primary's ability to write or serve traffic.
- Understanding it clarifies why quorum-based consensus matters
- Explains the need for fencing and STONITH mechanisms
- Highlights the cost of network partitions in distributed systems
- Connects directly to the CAP theorem trade-offs
AI Mentor Explanation
Imagine a cricket board's regional office loses phone contact with headquarters during a storm, so headquarters appoints a new regional secretary to keep issuing team selections, unaware the original secretary's line is merely down, not the secretary being unavailable. The original secretary, still fully operational, keeps announcing a different squad to local media. Once phone lines are restored, two conflicting squad announcements exist and officials must untangle which selections actually stand, exactly like reconciling two divergent write histories after a split-brain event.
Step-by-Step Explanation
Step 1
Network partition occurs
A network fault isolates the primary from the rest of the cluster, but the primary itself keeps running.
Step 2
The remaining nodes elect a new primary
Unable to reach the original primary, the majority partition promotes a replica to primary.
Step 3
Both nodes accept writes independently
The original primary, unaware it has been replaced, keeps accepting client writes, creating two diverging data histories.
Step 4
Partition heals and conflicts surface
When connectivity is restored, the cluster must detect the conflicting histories, fence one side, and reconcile or discard divergent writes.
What Interviewer Expects
- Clear explanation of how a network partition causes two active primaries
- Understanding of quorum-based consensus as the primary prevention mechanism
- Knowledge of fencing/STONITH to forcibly stop the old primary
- Connection to the CAP theorem and consistency-availability trade-offs
Common Mistakes
- Confusing split-brain with simple replication lag
- Not mentioning quorum or majority-based election as the fix
- Forgetting that reconciliation after a split-brain can lose data
- Failing to connect the problem to network partitions specifically
Best Answer (HR Friendly)
“Split-brain happens when a network problem causes two database nodes to each think they are the one true primary and both start accepting writes independently. When the network heals, you end up with two conflicting versions of the data that have to be reconciled, sometimes losing data in the process. The fix is requiring a majority of nodes (quorum) to agree before promoting a new primary, and forcibly shutting off the old primary once a new one takes over.”
Code Example
-- Pseudocode representing a quorum-based promotion guard,
-- the standard defense against split-brain.
function attemptPromotion(clusterSize, reachableNodes) {
const quorum = Math.floor(clusterSize / 2) + 1;
if (reachableNodes.length >= quorum) {
promoteToPrimary(); -- safe: this side has the majority
} else {
stepDownOrRefuseWrites(); -- minority side must NOT accept writes
}
}
-- Without this check, a minority-side node might promote itself,
-- resulting in two writable primaries: the split-brain scenario.Follow-up Questions
- How does quorum-based consensus (e.g. Raft, Paxos) prevent split-brain?
- What is fencing or STONITH, and how does it stop a rogue old primary?
- How does the CAP theorem relate to the split-brain trade-off?
- What manual reconciliation steps might be needed after a split-brain event resolves?
MCQ Practice
1. What condition typically triggers a split-brain scenario?
Split-brain arises when a network partition separates the primary from the cluster, but the primary keeps running and accepting writes unaware it has been replaced.
2. Which mechanism is the standard defense against split-brain?
Requiring a majority (quorum) of nodes to agree before electing a new primary prevents a minority partition from also promoting itself.
3. What is the purpose of fencing (STONITH) in preventing split-brain?
Fencing forcibly isolates or powers down the old primary so it cannot continue accepting writes after a new primary has been promoted.
Flash Cards
What is split-brain? — Two nodes both believing they are primary and independently accepting writes after a network partition.
What causes split-brain? — A network partition isolating the primary while it stays alive and unaware of a new promotion.
How is split-brain prevented? — Quorum-based consensus requiring a majority of nodes to agree before promoting a new primary.
What is fencing (STONITH)? — Forcibly cutting off the old primary’s ability to write, preventing dual-primary conflicts.