What is the purpose of the __consumer_offsets topic in Kafka?
Learn what Kafka's __consumer_offsets topic does, how it stores committed offsets per consumer group, why it is log-compacted, and common interview questions.
Expected Interview Answer
__consumer_offsets is a special internal, compacted Kafka topic that stores the committed offset (the position) of every consumer group for each partition it reads, so consumers can resume exactly where they left off after a restart or rebalance.
When a consumer commits, the broker writes a record keyed by (group, topic, partition) with the committed offset to __consumer_offsets rather than to ZooKeeper. Because the topic is log-compacted, only the latest offset per key is retained, keeping it small. It defaults to 50 partitions and a replication factor of 3 in production, and the group's coordinator broker owns the partition that holds a given group's offsets. It also stores consumer group metadata used during rebalances.
- Lets consumers resume from the last committed position after crashes or rebalances
- Removes the old ZooKeeper dependency for offset storage
- Log compaction keeps only the newest offset per key, so it stays compact
- Replicated and durable, so offset state survives broker failure
- Centralizes group membership and coordinator metadata
AI Mentor Explanation
Think of __consumer_offsets as the official scorer's ball-by-ball ledger. When a fresh scorer takes over mid-innings, they read the last recorded over and ball number and continue from precisely there rather than restarting the count. Kafka's offset topic is that ledger: each consumer group's last processed position is written down so a replacement instance resumes at the exact next delivery.
Step-by-Step Explanation
Step 1
Consumer reads records
A consumer in a group polls messages from assigned partitions and processes them.
Step 2
Commit the offset
The consumer commits the next offset to read, automatically or manually, to the group coordinator broker.
Step 3
Write to __consumer_offsets
The coordinator appends a record keyed by (group, topic, partition) to the internal offsets topic.
Step 4
Compaction retains latest
Log compaction discards older offset records for a key, keeping only the newest committed position.
Step 5
Resume on restart or rebalance
A restarting or reassigned consumer reads its group's committed offset and continues from the next record.
What Interviewer Expects
- Knowing it is an internal, log-compacted topic, not something you produce to directly
- That it replaced ZooKeeper for offset storage in modern Kafka
- Offsets are keyed per group, topic and partition
- Awareness of the group coordinator owning a group's offset partition
- Default partition count (50) and replication factor considerations
Common Mistakes
- Saying offsets are still stored in ZooKeeper in current Kafka versions
- Confusing the committed offset with the current log-end offset
- Thinking you should write to __consumer_offsets manually
- Assuming offsets are per-consumer rather than per-consumer-group
Best Answer (HR Friendly)
“It is a built-in Kafka topic that remembers how far each group of readers has gotten through the data. So if a reader restarts, it knows exactly where to continue instead of reprocessing everything or skipping messages.”
Code Example
# Show committed offset, log-end offset and lag per partition
kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe --group orders-service
# GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
# orders-service orders 0 10420 10420 0
# orders-service orders 1 9980 10005 25Follow-up Questions
- What is the difference between auto-commit and manual commit of offsets?
- How does log compaction keep __consumer_offsets from growing without bound?
- What happens to offsets if a consumer group is idle past offsets.retention.minutes?
- How does the group coordinator decide which broker holds a group's offsets?
- What is the difference between committed offset and current position?
MCQ Practice
1. How does Kafka keep the __consumer_offsets topic from growing indefinitely?
The topic is log-compacted, so only the most recent record for each (group, topic, partition) key is retained.
2. What replaced ZooKeeper as the store for consumer offsets in modern Kafka?
Committed offsets are written to the internal __consumer_offsets topic managed by the group coordinator.
3. Offset records in __consumer_offsets are keyed by which combination?
Each committed offset is stored per consumer group, per topic, per partition.
Flash Cards
What is __consumer_offsets? — An internal, log-compacted Kafka topic storing each consumer group's committed offset per topic-partition.
Why is it compacted? — So only the latest offset per (group, topic, partition) key is kept, keeping the topic small.
What did it replace? — ZooKeeper-based offset storage used in older Kafka versions.
Who writes to it? — The group coordinator broker, when a consumer commits its offset.