How does a Kafka consumer commit offsets and what is the risk of auto-commit?
How Kafka consumers commit offsets, why auto-commit can cause message loss or duplication, and how manual commitSync gives safer at-least-once delivery.
Expected Interview Answer
A Kafka consumer commits offsets to record how far it has read in each partition, storing the next offset to consume in the internal __consumer_offsets topic; with enable.auto.commit=true the client commits periodically in the background, and the risk is that these commits happen on a timer rather than after successful processing, which can cause message loss or duplication on failure.
When auto-commit is on, the consumer commits the current position roughly every auto.commit.interval.ms during poll calls, regardless of whether the polled records were actually processed. If the consumer crashes after an auto-commit but before finishing the work, those messages are marked consumed and are lost; if it crashes after processing but before the next commit, they are reprocessed, causing duplicates. Manual commits with commitSync or commitAsync after processing give at-least-once semantics by tying the commit to completed work, and pairing manual commits with idempotent processing or transactions moves you toward exactly-once.
- Offsets let a consumer resume exactly where it left off after a restart
- Manual commit ties progress to actual processing completion
- commitSync retries and blocks for reliability; commitAsync is faster
- Disabling auto-commit prevents silent message loss
- Enables at-least-once and, with idempotency, exactly-once delivery
AI Mentor Explanation
Imagine a scorer who marks a bookmark at the last ball fully recorded. If the scorer moves the bookmark forward every two minutes on a clock instead of after each ball is written down, a power cut can leave the bookmark ahead of the real work — those balls look scored but were never written. Auto-commit is that timer-based bookmark; committing only after each ball is actually recorded is the safe manual commit.
Step-by-Step Explanation
Step 1
Read messages with poll
The consumer calls poll() to fetch batches of records from its assigned partitions, tracking a current position per partition.
Step 2
Process each record
Do the actual work — write to a database, call a service, emit an event — before deciding the message is truly consumed.
Step 3
Commit the offset
Commit the offset of the next record to read to __consumer_offsets, either automatically on a timer or manually after processing.
Step 4
Prefer manual commit for safety
Set enable.auto.commit=false and call commitSync() (blocking, retried) or commitAsync() (non-blocking) once processing succeeds.
Step 5
Handle rebalances and restarts
On restart or rebalance, the consumer resumes from the last committed offset, so accurate commits determine what is replayed or skipped.
What Interviewer Expects
- Knowing offsets are stored in the __consumer_offsets topic
- Explaining that auto-commit is timer-driven, not processing-driven
- Describing message loss and duplication scenarios clearly
- Contrasting commitSync and commitAsync
- Linking manual commit to at-least-once and exactly-once semantics
Common Mistakes
- Saying offsets are stored on the consumer rather than in a Kafka topic
- Believing auto-commit commits only after messages are processed
- Confusing the committed offset with the current position
- Ignoring duplicates and assuming manual commit gives exactly-once by itself
- Using commitAsync without handling failed commits or ordering
Best Answer (HR Friendly)
“A Kafka consumer keeps a bookmark, called an offset, showing how far it has read so it can resume after a restart. Automatic bookmarking updates on a timer rather than after the work is done, so if the consumer crashes at the wrong moment it can skip messages or process them twice — committing manually after finishing the work avoids that.”
Code Example
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "orders-consumer");
props.put("enable.auto.commit", "false"); // disable timer-based commit
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("orders"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> record : records) {
process(record); // do the real work first
}
consumer.commitSync(); // commit only after successful processing
}
}Follow-up Questions
- What is the difference between commitSync and commitAsync?
- Where are consumer offsets stored and how are they replicated?
- How do you achieve exactly-once processing in Kafka?
- What is the difference between the committed offset and the current position?
- How does auto.commit.interval.ms affect the size of the loss or duplication window?
MCQ Practice
1. Where does a Kafka consumer store its committed offsets by default?
Modern Kafka stores committed offsets in the internal __consumer_offsets topic, keyed by group, topic and partition.
2. With enable.auto.commit=true, when are offsets committed?
Auto-commit advances offsets roughly every auto.commit.interval.ms regardless of whether processing finished, which is the source of its risk.
3. Which delivery guarantee does committing manually after processing provide by itself?
Committing after successful processing yields at-least-once; exactly-once additionally needs idempotency or transactions.
Flash Cards
What is a Kafka consumer offset? — The position marking the next message to read in a partition, committed so the consumer can resume after restart.
Where are offsets stored? — In the internal __consumer_offsets topic, keyed by group, topic and partition.
Why is auto-commit risky? — It commits on a timer, not after processing, so a crash can cause message loss or duplication.
commitSync vs commitAsync? — commitSync blocks and retries for reliability; commitAsync is non-blocking and faster but does not retry on failure.
Continue Learning
Related Interview Questions
What is consumer rebalancing in Kafka and why can it be disruptive?
hard
What is the difference between Kafka and a traditional message queue?
medium
What is a Kafka partition and why is it the unit of parallelism?
medium
What are the different delivery semantics in Kafka (at-most-once, at-least-once, exactly-once)?
hard