What is consumer rebalancing in Kafka and why can it be disruptive?
What Kafka consumer rebalancing is, how the coordinator reassigns partitions, and why it pauses processing, causes duplicates, and stalls groups.
Expected Interview Answer
Consumer rebalancing is the process by which Kafka redistributes a topic's partitions across the consumers in a group whenever membership or assignment changes — such as a consumer joining, leaving, crashing, or new partitions being added — so that every partition has exactly one active consumer in the group.
It is coordinated by the group coordinator broker: with the classic eager protocol, all consumers revoke their partitions and stop consuming while a new assignment is computed, causing a stop-the-world pause; the incremental cooperative protocol reduces this by only moving the partitions that actually need to change. Rebalances are disruptive because processing halts on affected partitions, in-flight work can be lost or reprocessed if offsets were not committed, local state and caches tied to partitions must be rebuilt, and frequent triggers — often from slow processing exceeding max.poll.interval.ms — can cause repeated rebalance storms that stall the group.
- Automatically balances load across available consumers
- Provides fault tolerance when a consumer dies
- Lets a group scale out by adding consumers up to the partition count
- Cooperative rebalancing minimizes stop-the-world pauses
- Ensures each partition always has exactly one owner in the group
AI Mentor Explanation
Think of fielders each covering set zones of the outfield. When one player leaves injured or a new one runs on, the captain pauses play to reassign every zone so no gap is left uncovered. In eager mode the whole field stops until positions are settled; if it happens too often, the side spends more time reshuffling than actually fielding. Kafka rebalancing reassigns partitions to consumers exactly like reallocating those fielding zones.
Step-by-Step Explanation
Step 1
A trigger occurs
A consumer joins, leaves, crashes, misses a heartbeat, or partitions are added — any of which changes group membership or the partition set.
Step 2
Coordinator detects the change
The group coordinator broker notices via heartbeats or a join request and initiates a rebalance for the group.
Step 3
Partitions are revoked
In the eager protocol all consumers revoke their partitions and pause; in cooperative mode only the partitions that must move are revoked.
Step 4
New assignment is computed
A partition assignor (range, round-robin, sticky, or cooperative-sticky) computes which consumer owns which partition.
Step 5
Consumers resume
Each consumer receives its assignment, seeks to the last committed offset, and resumes consuming — replaying any uncommitted work.
What Interviewer Expects
- Defining rebalancing as partition redistribution across a consumer group
- Explaining the role of the group coordinator and heartbeats
- Contrasting eager stop-the-world with cooperative incremental rebalancing
- Identifying max.poll.interval.ms and slow processing as common triggers
- Describing state loss, duplicates, and rebalance storms as costs
Common Mistakes
- Thinking a rebalance moves data rather than partition ownership
- Assuming rebalancing never pauses consumption
- Ignoring that slow processing past max.poll.interval.ms triggers a rebalance
- Not committing offsets before revocation, causing avoidable reprocessing
- Believing more consumers than partitions increases throughput
Best Answer (HR Friendly)
“When consumers share the work of reading a Kafka topic, each one handles a slice of it. If a consumer joins or drops out, Kafka reshuffles who handles which slice — that is rebalancing. It can be disruptive because reading pauses during the reshuffle, some work may be redone, and if it happens too often the whole group spends its time reorganizing instead of processing.”
Code Example
consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
// Commit progress before losing these partitions to avoid reprocessing
consumer.commitSync(currentOffsets);
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
// Rebuild any partition-local state here
}
});
// Reduce rebalance pauses with the cooperative assignor
props.put("partition.assignment.strategy",
"org.apache.kafka.clients.consumer.CooperativeStickyAssignor");Follow-up Questions
- How does the CooperativeStickyAssignor reduce rebalance impact?
- What settings cause a consumer to be kicked from a group and trigger a rebalance?
- How do static group membership (group.instance.id) and reduced rebalances relate?
- What is a rebalance storm and how do you diagnose one?
- How does max.poll.interval.ms differ from session.timeout.ms?
MCQ Practice
1. What does a Kafka consumer rebalance redistribute?
A rebalance reassigns which consumer in the group owns which partition; it does not move the underlying data.
2. Which is a common cause of unexpected rebalances?
If a consumer takes too long between poll calls, the coordinator considers it dead and triggers a rebalance.
3. Which protocol avoids a full stop-the-world pause during rebalancing?
Cooperative incremental rebalancing revokes only the partitions that must move, so unaffected consumers keep working.
Flash Cards
What is consumer rebalancing? — Redistribution of a topic's partitions across the consumers in a group when membership or assignment changes.
Who coordinates a rebalance? — The group coordinator broker, using heartbeats and join requests to detect changes.
Why is eager rebalancing disruptive? — All consumers revoke partitions and pause until a new assignment is computed — a stop-the-world halt.
What commonly triggers surprise rebalances? — Slow processing exceeding max.poll.interval.ms, making the coordinator treat the consumer as failed.
Continue Learning
Related Interview Questions
What is a Kafka partition and why is it the unit of parallelism?
medium
How does a Kafka consumer commit offsets and what is the risk of auto-commit?
medium
How does cooperative incremental rebalancing differ from eager rebalancing in Kafka?
hard
What is the difference between Kafka and a traditional message queue?
medium