How does Kafka achieve durability with replication and the ISR?
Learn how Kafka achieves durability using partition replication, the in-sync replica set, acks=all, and min.insync.replicas to prevent data loss.
Expected Interview Answer
Kafka achieves durability by replicating each partition across multiple brokers and only acknowledging writes once they reach the in-sync replicas (ISR) — the set of replicas that are fully caught up with the leader.
Every partition has one leader and several followers; the ISR is the subset of replicas whose fetch position is within replica.lag.time.max.ms of the leader. With acks=all and min.insync.replicas set (e.g. 2), a produce request is acknowledged only after all in-sync replicas persist it, so the data survives a broker failure. If a leader dies, a new leader is elected from the ISR, preserving all acknowledged records. Setting min.insync.replicas above 1 and disabling unclean leader election trades some availability for a guarantee that no acknowledged write is ever lost.
- Survives broker failures without data loss
- Fast failover via ISR-based leader election
- Tunable durability through acks and min.insync.replicas
- Prevents silent loss when unclean election is disabled
- Balances consistency against availability per topic
AI Mentor Explanation
The leader is the official scorebook, and each in-sync replica is an assistant scorer copying every ball in real time. A run is only declared confirmed once the required number of assistants have recorded it too — that is acks=all with min.insync.replicas. If the head scorer collapses, a fully caught-up assistant takes over as the official book, and no confirmed run is lost. An assistant who falls behind is dropped from the trusted set, just as a lagging replica leaves the ISR.
Step-by-Step Explanation
Step 1
Replicate each partition
Set a replication factor (e.g. 3) so every partition has a leader and follower copies on different brokers.
Step 2
Track the ISR
Followers within replica.lag.time.max.ms of the leader form the in-sync replica set.
Step 3
Require strong acks
Producers set acks=all so a write is acknowledged only after all in-sync replicas persist it.
Step 4
Set min.insync.replicas
Require at least N in-sync replicas (e.g. 2); if fewer, the write is rejected rather than risked.
Step 5
Disable unclean leader election
Keep unclean.leader.election.enable=false so only an in-sync replica can become leader, preventing lost writes.
What Interviewer Expects
- Leader/follower replication model
- Definition of the ISR and how replicas leave it
- Interaction of acks=all and min.insync.replicas
- What unclean leader election risks
- The availability vs durability trade-off
Common Mistakes
- Thinking replication alone guarantees no loss without acks=all
- Ignoring min.insync.replicas and still expecting durability
- Leaving unclean leader election enabled in critical topics
- Assuming a follower is always in the ISR regardless of lag
Best Answer (HR Friendly)
“Kafka keeps several copies of each partition on different servers. A write is only confirmed once the up-to-date copies have safely stored it, so if one server fails, another current copy takes over without losing any confirmed data. You tune how many copies must agree to balance safety against availability.”
Code Example
# Create a topic with 3 replicas and require 2 in-sync
kafka-topics.sh --create --topic payments \
--partitions 6 --replication-factor 3 \
--config min.insync.replicas=2 \
--config unclean.leader.election.enable=false \
--bootstrap-server localhost:9092
# Producer must wait for all in-sync replicas
# acks=all + min.insync.replicas=2 => no acknowledged write is lost
kafka-console-producer.sh --topic payments \
--producer-property acks=all \
--bootstrap-server localhost:9092Follow-up Questions
- What happens to writes when the ISR shrinks below min.insync.replicas?
- How does unclean leader election cause data loss?
- How is a replica added back into the ISR after catching up?
- Why is replication factor 3 a common production choice?
- How do acks=1 and acks=all differ in durability?
MCQ Practice
1. What is the in-sync replica (ISR) set?
The ISR is the set of replicas whose fetch position is within the lag threshold of the leader.
2. With acks=all and min.insync.replicas=2, when is a write acknowledged?
The write is acknowledged only once the required number of in-sync replicas have persisted it, ensuring durability.
3. Why disable unclean leader election for critical topics?
Disabling it ensures only in-sync replicas can become leader, so acknowledged records are never lost.
Flash Cards
What is the ISR? — The in-sync replica set — followers caught up with the leader within replica.lag.time.max.ms.
Role of min.insync.replicas? — Minimum in-sync replicas that must persist a write (with acks=all) before it is acknowledged.
Danger of unclean leader election? — An out-of-sync replica can become leader and drop acknowledged writes — disable it for durability.
Durability vs availability? — Higher min.insync.replicas means safer writes but rejects them when too few replicas are in sync.
Continue Learning
Related Interview Questions
What is the difference between a leader and follower replica in Kafka?
medium
How does replication factor and min.insync.replicas affect availability in Kafka?
hard
What is a Kafka broker and how does a Kafka cluster work?
medium
What is acks configuration in a Kafka producer and what do 0, 1, and all mean?
medium