What is a Kafka offset and how is consumer position tracked?
Understand what a Kafka offset is, how consumers commit position, where offsets are stored and how commit timing controls delivery semantics, with examples.
Expected Interview Answer
A Kafka offset is a monotonically increasing integer that uniquely identifies the position of each record within a partition, and a consumer's progress is tracked by committing the offset of the last record it has processed.
Offsets are assigned per partition, so the same offset number can exist in different partitions. Consumers do not delete messages when they read them; instead they remember their position by committing offsets, typically to the internal __consumer_offsets topic keyed by group, topic, and partition. On restart or rebalance, a consumer resumes from its last committed offset. Commits can be automatic on a timer or manual, and the choice between committing before or after processing determines whether failures cause message loss or reprocessing.
- Lets consumers resume exactly where they left off
- Decouples message retention from consumption
- Enables replay by resetting to an earlier offset
- Supports multiple independent groups on one topic
- Gives control over at-least-once vs at-most-once semantics
AI Mentor Explanation
An offset is like the ball number in a scorebook: every delivery in an over is numbered in sequence so you know exactly which ball you are on. If the match pauses, the scorer notes the last ball recorded and resumes from the next one, never re-scoring what is already logged. Kafka does the same by committing the last processed offset so a consumer restarts from precisely the right delivery rather than the start of the innings.
Step-by-Step Explanation
Step 1
Records get sequential offsets
As messages arrive in a partition, Kafka assigns each an increasing offset starting at zero.
Step 2
Consumer reads and processes
The consumer polls records from its assigned partitions and processes them in offset order.
Step 3
Commit the position
It commits the offset of the last processed record, usually to the __consumer_offsets topic keyed by group, topic, and partition.
Step 4
Resume from commit
After a restart or rebalance, the consumer reads its committed offset and continues from the next record.
Step 5
Choose commit strategy
Auto-commit on a timer is simplest; manual commit after processing gives at-least-once, before processing gives at-most-once.
What Interviewer Expects
- Offset as a per-partition sequential position, not global
- That reading does not delete messages
- Where offsets are stored (__consumer_offsets, per group)
- How commit timing affects delivery semantics
- How offset reset enables replay
Common Mistakes
- Thinking offsets are unique across the whole topic
- Believing consuming a message removes it from Kafka
- Confusing committed offset with current read position
- Ignoring that auto-commit can cause loss or duplicates
- Assuming all consumer groups share one offset
Best Answer (HR Friendly)
“A Kafka offset is just a numbered position for each message inside a partition, like a page number in a book. Consumers remember the last position they finished by committing it, so if they restart they pick up right where they stopped instead of reading everything again.”
Code Example
props.put("enable.auto.commit", "false");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("payments"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> record : records) {
process(record.value()); // do the work first
}
consumer.commitSync(); // commit only after success => at-least-once
}Follow-up Questions
- What is the difference between auto-commit and manual commit?
- How do you achieve exactly-once processing in Kafka?
- How do you reset a consumer group to reprocess old messages?
- Where are consumer offsets stored?
- What is the difference between the current offset and the committed offset?
MCQ Practice
1. A Kafka offset uniquely identifies a record within?
Offsets are assigned per partition, so the same offset number can appear in different partitions of the same topic.
2. Where are consumer group offsets stored by default in modern Kafka?
Kafka stores committed offsets in the internal __consumer_offsets topic, keyed by group, topic, and partition.
3. Committing an offset before processing a record risks?
Committing before processing yields at-most-once semantics: a failure after the commit means the record is skipped and effectively lost.
Flash Cards
What is a Kafka offset? — A sequential integer identifying a record's position within a partition.
Is an offset unique across a topic? — No; offsets are per partition, so the same number can exist in different partitions.
Where are committed offsets stored? — In the internal __consumer_offsets topic, keyed by group, topic, and partition.
Commit after vs before processing? — After = at-least-once (possible duplicates); before = at-most-once (possible loss).