Why Kafka Replicates Partitions
Every partition in a Kafka topic can have multiple replicas spread across different brokers, controlled by the replication.factor topic setting. One replica is designated the leader, and all producer writes and consumer reads for that partition go through the leader; the remaining replicas are followers that passively copy the leader's log. Replication exists purely for durability and availability: if the broker hosting the leader crashes, a follower that was fully caught up can be promoted to leader with no data loss for acknowledged writes.
Cricket analogy: Think of a partition's replicas like a cricket team carrying both an opening batsman and a designated backup opener in the squad; if Rohit Sharma gets injured before a Test at Lord's, the backup steps in immediately so the innings doesn't stall, just as a follower replica steps in when the leader broker dies.
Leaders, Followers, and the In-Sync Replica Set
The in-sync replica set (ISR) is the subset of a partition's replicas that have fetched up to the leader's latest offset within replica.lag.time.max.ms. Followers pull data from the leader via fetch requests, the same mechanism consumers use, rather than the leader pushing data out. A replica falls out of the ISR if it stops fetching or falls too far behind, and it can rejoin once it catches back up; only ISR members are eligible to become leader without data loss when min.insync.replicas and acks=all are configured together.
Cricket analogy: The ISR is like the fielders currently inside the 30-yard circle during powerplay overs — only those close enough and positioned correctly, like Jonty Rhodes at cover, are eligible to make the crucial diving stop the captain relies on.
Leader Election
When a leader fails, the controller broker chooses a new leader from the ISR, preferring the first replica in the ISR list to minimize data loss. If unclean.leader.election.enable is turned on, Kafka may promote an out-of-sync replica when no ISR member is available, trading availability for potential data loss. Producers configured with acks=all only receive an acknowledgment once the message is written to the leader and replicated to all current ISR members, which is what makes acks=all safe against leader failover.
Cricket analogy: When a captain is ruled out injured before a match, the selectors promote the vice-captain who has been playing every match in the current series, not a player who missed the last three fixtures, just as Kafka prefers an ISR member for leader election.
# Inspect a topic's current leader, replicas, and ISR
kafka-topics.sh --describe --topic orders --bootstrap-server broker1:9092
# Example output
# Topic: orders Partition: 0 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3
# Topic: orders Partition: 1 Leader: 1 Replicas: 1,3,2 Isr: 1,3
# Producer config for maximum durability
# acks=all
# min.insync.replicas=2 (set on the topic or broker)Handling Leader Failures and Preferred Leaders
Kafka periodically runs preferred leader election, moving leadership back to the 'preferred leader' (the first replica listed when the partition was created) after a broker that was leader recovers, to keep leadership balanced across the cluster. Without this rebalancing, leadership can pile up on a few brokers after rolling restarts, causing uneven CPU and network load. Tools like kafka-leader-election.sh or Cruise Control automate this so operators don't have to manually trigger PreferredReplicaLeaderElection after every deployment.
Cricket analogy: After an injury forces a temporary vice-captain to lead for a few overs, the team reverts captaincy to the original captain once fit, keeping leadership consistent rather than letting it drift permanently, like Kafka's preferred leader election restoring balance.
Setting acks=1 (leader-only acknowledgment) or enabling unclean.leader.election.enable=true trades durability for availability — a leader crash before followers replicate a message, or an unclean election promoting a lagging replica, can silently lose acknowledged data. Use acks=all with min.insync.replicas>=2 for critical data.
- Each partition has one leader (handles all reads/writes) and follower replicas that replicate via pull-based fetch requests.
- The ISR tracks replicas caught up within replica.lag.time.max.ms; only ISR members are safe leader candidates by default.
- acks=all plus min.insync.replicas>=2 guarantees an acknowledged write survives a leader failure without data loss.
- unclean.leader.election.enable=true allows promoting an out-of-sync replica during outages, risking data loss for availability.
- The controller performs leader election and propagates the new leader via LeaderAndIsrRequest to the cluster.
- Preferred leader election periodically rebalances leadership back to each partition's originally assigned first replica.
Practice what you learned
1. What does a Kafka follower replica do to stay in sync with the leader?
2. A replica is removed from the ISR when...
3. What guarantee does acks=all provide when paired with min.insync.replicas=2?
4. What is the risk of enabling unclean.leader.election.enable=true?
5. What is the purpose of Kafka's preferred leader election?
Was this page helpful?
You May Also Like
Log Compaction
How Kafka's compaction cleanup policy retains only the latest value per key instead of expiring records by time.
The Kafka Controller
The broker role responsible for cluster-wide coordination: leader election, metadata propagation, and admin operations.
Retention Policies in Kafka
How Kafka's time- and size-based retention deletes old log segments, and how it interacts with consumer lag.
ZooKeeper vs KRaft
How Kafka's metadata layer evolved from an external ZooKeeper ensemble to the built-in, Raft-based KRaft consensus protocol.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics