How does Kafka handle backpressure and slow consumers?
Learn how Kafka's pull model, consumer lag and poll tuning handle backpressure and slow consumers, with tactics, code and common interview questions.
Expected Interview Answer
Kafka handles backpressure through its pull-based consumer model: consumers fetch messages at their own pace, so a slow consumer simply lags behind rather than being overwhelmed, while the broker retains data on disk up to the configured retention and lets each consumer track its own offset.
Because consumers pull with fetch requests bounded by max.poll.records, fetch.max.bytes and max.partition.fetch.bytes, they never receive more than they ask for. A slow consumer accumulates consumer lag (the gap between the log-end offset and its committed offset), which is the primary signal to watch. If processing per poll takes longer than max.poll.interval.ms the consumer is evicted and a rebalance reassigns its partitions. You relieve pressure by scaling out the consumer group up to the partition count, tuning poll batch sizes, pausing partitions, or offloading work asynchronously.
- Pull model means consumers are never flooded beyond what they request
- Broker retention decouples producer speed from consumer speed
- Consumer lag gives a precise, monitorable backpressure signal
- Scaling the consumer group parallelizes processing across partitions
- pause/resume lets an app throttle specific partitions without dropping data
AI Mentor Explanation
Think of a bowling machine that only fires the next ball when the batter taps a pedal. A tired batter simply taps slower and the machine waits, holding a hopper of balls in reserve. Kafka's pull model works the same way: the consumer requests each batch when ready, the broker keeps the backlog on disk, and a slow consumer just falls further behind in overs rather than being bombarded with deliveries it cannot face.
Step-by-Step Explanation
Step 1
Pull, don't push
Consumers issue fetch requests, so they receive only what they explicitly ask for each poll.
Step 2
Bound the batch
max.poll.records, fetch.max.bytes and max.partition.fetch.bytes cap how much a single poll returns.
Step 3
Let lag absorb the mismatch
The broker retains data on disk, so a slow consumer simply builds consumer lag instead of losing messages.
Step 4
Watch max.poll.interval.ms
If processing between polls exceeds this, the consumer is evicted and its partitions are rebalanced away.
Step 5
Relieve the pressure
Scale the group up to partition count, pause/resume partitions, or offload heavy work asynchronously.
What Interviewer Expects
- Recognition that Kafka is pull-based, not push-based
- Consumer lag defined as log-end offset minus committed offset
- Awareness of max.poll.records and max.poll.interval.ms tuning
- That scaling is bounded by the number of partitions
- Practical relief tactics: pause/resume, async processing, adding consumers
Common Mistakes
- Claiming Kafka pushes messages to consumers and can overwhelm them
- Confusing consumer lag with broker disk being full
- Thinking you can scale a group beyond the partition count for more parallelism
- Ignoring max.poll.interval.ms and causing repeated rebalances
- Assuming slow consumers lose messages rather than just falling behind
Best Answer (HR Friendly)
“Kafka lets each reader pull messages at its own speed rather than having data forced onto it, so a slow reader just falls behind instead of crashing. Nothing is lost because the messages stay stored on disk, and teams watch a number called consumer lag to know when to add more readers or speed up processing.”
Code Example
props.put("max.poll.records", 100);
props.put("max.partition.fetch.bytes", 1048576);
props.put("max.poll.interval.ms", 300000);
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"));
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> r : records) {
process(r); // if this is slow, lag grows but nothing is lost
}
if (downstreamOverloaded()) {
consumer.pause(consumer.assignment()); // throttle without leaving the group
} else {
consumer.resume(consumer.assignment());
}
consumer.commitSync();
}Follow-up Questions
- How exactly do you calculate consumer lag and what tools expose it?
- What happens during a rebalance triggered by exceeding max.poll.interval.ms?
- Why can't you scale a consumer group beyond the number of partitions?
- When would you use pause() and resume() instead of just slowing the poll loop?
- How does log retention configuration interact with a persistently slow consumer?
MCQ Practice
1. Why does Kafka's design naturally avoid overwhelming slow consumers?
Kafka consumers pull with bounded fetch requests, so they only receive what they ask for and slow consumers simply lag.
2. What does consumer lag measure?
Lag is how far a consumer's committed offset trails the latest offset in the partition — the key backpressure signal.
3. Exceeding which setting causes a slow consumer to be evicted and rebalanced?
If the gap between poll() calls exceeds max.poll.interval.ms, the coordinator considers the consumer dead and reassigns its partitions.
Flash Cards
Push or pull? — Kafka consumers pull, requesting bounded batches, so they are never flooded.
Consumer lag? — Log-end offset minus the consumer's committed offset — how far behind it is.
What caps a single poll? — max.poll.records, fetch.max.bytes and max.partition.fetch.bytes.
Max parallelism of a group? — Bounded by the number of partitions on the topic.
Throttle without leaving the group? — Use consumer.pause() and consumer.resume() on assigned partitions.