Broker-Level Failures
The most disruptive broker-level failure is losing the controller broker, the single broker responsible for managing partition leader elections and cluster metadata changes; while a new controller election happens automatically within seconds via ZooKeeper or the KRaft quorum, any partition leader elections that were mid-flight during the controller failure can leave affected partitions briefly without a leader, causing producers and consumers targeting those partitions to receive NOT_LEADER_FOR_PARTITION errors until election completes.
Cricket analogy: It's like losing the third umpire mid-review during a crucial DRS decision; a replacement official steps in quickly, but any review that was actively being processed at that exact moment leaves both teams briefly stuck not knowing the outcome.
A slower but equally dangerous failure mode is disk I/O degradation on a broker without full failure: a broker with a failing but not-yet-dead disk can respond to health checks while its actual write latency spikes into hundreds of milliseconds, causing it to fall behind on replication (becoming under-replicated) or even get kicked out of the in-sync replica set (ISR) by the leader, which is often harder to diagnose than a clean crash because the broker process itself never stops running.
Cricket analogy: It's like a fast bowler whose action has quietly degraded due to a hidden knee issue; he's still bowling every over and technically fit to field, but his pace has dropped from 140 to 120 km/h, which is far harder for the team physio to catch than an outright injury that forces him off.
# Check which replicas have fallen out of the in-sync replica set
kafka-topics.sh --bootstrap-server broker1:9092 --describe --topic orders
# Look for Isr: list shorter than Replicas: list
# Example output showing a broker (id=3) dropped from ISR
# Topic: orders Partition: 2 Leader: 1 Replicas: 1,2,3 Isr: 1,2
# Check for slow disk I/O correlated with the drop
iostat -x 1 5
# Watch %util and await columns for the affected broker's data diskA broker with degrading but not fully failed disk I/O is one of the hardest failures to catch with basic health checks, since the process stays alive and responds to simple TCP or metadata requests; always alert on replica.lag.time.max.ms and ISR shrink events, not just process liveness.
Client-Side and Coordination Failures
Consumer group rebalancing storms are a common client-side failure mode: if a consumer's processing loop takes longer than max.poll.interval.ms to call poll() again (often because of a slow downstream call inside the message-processing logic), the group coordinator considers that consumer dead and triggers a rebalance, reassigning its partitions; if several consumers in the group are all borderline slow, this can cascade into a repeated rebalance loop where the group barely makes forward progress because it keeps reorganizing instead of processing.
Cricket analogy: It's like a fielding captain who keeps swapping fielders in and out of position because each one takes slightly too long to signal they're ready, and the constant reshuffling means the team spends more time repositioning than actually fielding the ball.
Split-brain and zombie producer scenarios occur when a producer or consumer appears dead to the coordinator due to a garbage collection pause or network partition, gets fenced out or has its partitions reassigned, but then resumes execution and continues sending requests as if nothing happened; without proper fencing via producer epochs (for transactional producers) or generation IDs (for consumer group members), this zombie instance can write stale or duplicate data even after a new instance has taken over its role.
Cricket analogy: It's like a batsman who gets given out but, due to a communication delay, keeps playing shots at the crease because he never received the umpire's signal, potentially scoring runs that shouldn't count since he was already ruled out of the game.
Tuning max.poll.records lower and offloading slow downstream work (like external API calls) to a separate thread pool outside the poll loop are the two most effective fixes for rebalance storms caused by exceeding max.poll.interval.ms.
- Controller failure triggers automatic re-election, but partitions mid-election at the failure moment briefly lose their leader.
- Degrading disk I/O on a live broker process is harder to detect than a crash since basic health checks still pass.
- ISR shrink events and replica.lag.time.max.ms should be monitored alongside raw broker process liveness.
- Exceeding max.poll.interval.ms triggers a consumer rebalance, and repeated near-misses cause cascading rebalance storms.
- Offloading slow downstream work outside the poll loop is a key fix for consumer rebalance storm issues.
- Zombie producers or consumers can resume sending stale requests after being fenced out due to GC pauses or network partitions.
- Producer epochs and consumer generation IDs are the fencing mechanisms that prevent zombie instances from corrupting data.
Practice what you learned
1. What is the risk when the controller broker fails, even though re-election happens automatically?
2. Why is degrading disk I/O on a broker often harder to diagnose than a full broker crash?
3. What typically triggers a consumer group rebalance storm?
4. What mechanism prevents a zombie transactional producer from writing stale data after being fenced out?
5. What is a primary fix for consumer group rebalance storms caused by slow processing?
Was this page helpful?
You May Also Like
Monitoring Kafka Clusters
The essential metrics, tools, and alerting strategies for keeping a production Kafka cluster healthy and observable.
Idempotent and Transactional Producers
How Kafka producers avoid duplicate writes and coordinate atomic multi-partition commits using idempotence and transactions.
Scaling Kafka Clusters
Strategies for growing Kafka capacity: adding brokers, reassigning partitions, and choosing the right partition count upfront.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics