What is read repair and how does anti-entropy work in Cassandra?
Understand Cassandra read repair vs anti-entropy: digest reads, last-write-wins, Merkle-tree repair, and why repair runs within gc_grace_seconds.
Expected Interview Answer
Read repair and anti-entropy are Cassandra's two mechanisms for keeping replicas consistent in an eventually consistent cluster. Read repair fixes stale replicas opportunistically during reads by comparing the responses (or their digests) from replicas and pushing the newest version to any that are behind, while anti-entropy repair is a proactive, background process that uses Merkle trees to find and reconcile all divergences between replicas, whether or not the data is being read.
During a read, the coordinator queries replicas per the consistency level; it requests full data from the fastest replica and digests from the others. If the digests disagree, Cassandra performs a foreground read repair — reading full data, resolving to the latest timestamp (last-write-wins), returning that to the client, and writing the reconciled value back to the out-of-date replicas. Because reads only touch data that is actually queried, cold or rarely-read data can stay divergent, so operators run anti-entropy repair with nodetool repair. That process builds Merkle trees (hash trees) of each replica's data ranges, compares them to pinpoint differing ranges without shipping all the data, and streams only the mismatched ranges to bring replicas into agreement. Running repair at least once per gc_grace_seconds is essential to avoid resurrected deletes.
- Keeps replicas consistent without blocking writes
- Read repair fixes hot data automatically as it is read
- Anti-entropy repair heals cold data that reads never touch
- Merkle trees compare replicas cheaply, streaming only mismatches
- Regular repair prevents deleted data from resurrecting
AI Mentor Explanation
Read repair is like two scorers checking each other's books only when someone actually asks for the current score: they spot a mismatch, agree on the later, authoritative entry, tell the enquirer, and quietly fix the wrong book. Anti-entropy is the end-of-day audit where officials hash-summarise whole sections of every scorebook, compare summaries to find exactly which overs differ, and recopy only those overs — so even pages nobody asked about all season are reconciled before the records are archived.
Step-by-Step Explanation
Step 1
Coordinator reads replicas
For a read at the given consistency level, the coordinator asks the fastest replica for full data and the others for digests.
Step 2
Compare digests
If all digests match, the value is returned; if they disagree, a mismatch is detected and read repair triggers.
Step 3
Resolve and return
Cassandra reads full data from the disagreeing replicas, resolves via last-write-wins timestamps, and returns the newest value to the client.
Step 4
Write back the fix
The reconciled value is written back to any out-of-date replicas, so hot data self-heals as it is read.
Step 5
Run anti-entropy repair
nodetool repair builds Merkle trees per range, compares them across replicas, and streams only the differing ranges to heal cold data.
What Interviewer Expects
- Distinction between opportunistic read repair and proactive anti-entropy repair
- How digest reads detect mismatches efficiently
- Last-write-wins conflict resolution by timestamp
- Role of Merkle trees in comparing replicas cheaply
- Why repair must run within gc_grace_seconds to avoid resurrected deletes
Common Mistakes
- Assuming read repair alone keeps all data consistent
- Never running nodetool repair, letting cold data diverge
- Not repairing within gc_grace_seconds, causing zombie data
- Confusing Merkle-tree repair with full data comparison
- Thinking higher consistency levels remove the need for anti-entropy repair
Best Answer (HR Friendly)
“Cassandra keeps its copies of data in sync in two ways: read repair fixes any out-of-date copy on the fly whenever that data is read, and anti-entropy repair is a scheduled background job that compares all the copies using compact fingerprints and patches the parts that differ. Together they make sure every server eventually holds the same, correct data, including data that is rarely read.”
Code Example
# Full repair of all keyspaces on this node's ranges
nodetool repair
# Repair a specific keyspace/table, primary ranges only (run on every node)
nodetool repair -pr my_keyspace my_table
# Incremental repair (default in modern Cassandra) marks repaired SSTables
nodetool repair --full # force a full, non-incremental repair when needed
# Key rule: run repair on every node at least once every gc_grace_seconds
# (default 10 days) to prevent deleted data from resurrecting.Follow-up Questions
- How does last-write-wins conflict resolution decide the winning value?
- What is the difference between foreground and background read repair?
- How do Merkle trees make repair efficient compared to full comparison?
- What is incremental repair and how does it differ from full repair?
- Why must repair complete within gc_grace_seconds?
MCQ Practice
1. What triggers a read repair in Cassandra?
During a read the coordinator compares digests; if they disagree it reconciles and writes the newest value back to stale replicas.
2. What data structure does anti-entropy repair use to find divergences cheaply?
Merkle trees hash data ranges so replicas can compare summaries and stream only the ranges that differ.
3. Why is running repair within gc_grace_seconds important?
If tombstones are purged before a lagging replica is repaired, that replica's old data reappears as zombie rows.
Flash Cards
What is read repair? — An opportunistic fix during reads: mismatched replicas are reconciled to the latest value and the stale ones are updated.
What is anti-entropy repair? — A proactive background process (nodetool repair) that uses Merkle trees to find and stream divergent ranges between replicas.
How does Cassandra resolve conflicts? — Last-write-wins by cell timestamp.
Why use Merkle trees? — They let replicas compare hash summaries and transfer only the ranges that differ, avoiding full-data comparison.
Repair and gc_grace_seconds? — Repair must run within gc_grace_seconds on every node to prevent deleted data from resurrecting.