What is a consumer group in Kafka and how does it enable scaling?
Learn what a Kafka consumer group is, how partitions map one-to-one to consumers, how it scales throughput and rebalances, with examples and interview Q&A.
Expected Interview Answer
A Kafka consumer group is a set of consumers that share a common group ID and cooperatively read from a topic, with each partition assigned to exactly one consumer in the group so the work is divided and processed in parallel.
Kafka distributes a topic's partitions across the members of a group, guaranteeing that no two consumers in the same group read the same partition, which preserves per-partition ordering while spreading load. Adding consumers scales throughput up to the number of partitions; beyond that, extra consumers sit idle. When a member joins or leaves, Kafka triggers a rebalance to reassign partitions. Different groups each get their own full copy of the stream, enabling multiple independent applications to consume the same topic.
- Parallel processing across partitions for higher throughput
- Automatic failover when a consumer dies
- Preserves per-partition message ordering
- Multiple groups can independently read the same topic
- Horizontal scaling by simply adding consumers
AI Mentor Explanation
A consumer group is like a fielding side splitting the boundary among fielders: each region of the rope is covered by exactly one fielder so no ball is chased by two players at once. Add fielders and you cover more ground, but you can never usefully post more fielders than there are regions to guard. If one fielder is injured, the captain reassigns their zone to another, just as Kafka rebalances a partition to a surviving consumer.
Step-by-Step Explanation
Step 1
Set a shared group ID
Every consumer that should cooperate configures the same group.id so Kafka treats them as one group.
Step 2
Partitions get assigned
The group coordinator distributes the topic's partitions so each partition is owned by exactly one member.
Step 3
Consume in parallel
Each member polls only its assigned partitions, processing records concurrently while preserving per-partition order.
Step 4
Rebalance on membership change
When a consumer joins or leaves, Kafka triggers a rebalance to redistribute partitions across the current members.
Step 5
Commit offsets per group
The group tracks its own committed offsets, so restarts resume where the group left off independently of other groups.
What Interviewer Expects
- One-partition-to-one-consumer rule within a group
- How adding consumers scales throughput up to the partition count
- That different groups each read the full stream independently
- Awareness of rebalancing and failover
- Why extra consumers beyond partitions stay idle
Common Mistakes
- Thinking two consumers in one group can read the same partition
- Believing you can scale past the number of partitions
- Confusing consumer groups with separate topics
- Ignoring that rebalances pause consumption briefly
- Assuming all groups share one offset instead of per-group offsets
Best Answer (HR Friendly)
“A Kafka consumer group is a team of workers that split a topic's partitions between them so each piece of data is handled by only one worker, which lets you process messages in parallel. Add more workers and you get more throughput, and if one fails, its work is automatically handed to another.”
Code Example
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "order-processors"); // same id = one group
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> record : records) {
System.out.printf("partition=%d offset=%d value=%s%n",
record.partition(), record.offset(), record.value());
}
}Follow-up Questions
- What happens during a consumer group rebalance?
- How does Kafka decide which consumer gets which partition?
- What is the effect of having more consumers than partitions?
- How do two different consumer groups reading the same topic differ?
- What are cooperative rebalancing and static group membership?
MCQ Practice
1. Within a single consumer group, a partition is read by how many consumers?
Kafka assigns each partition to exactly one consumer per group, which preserves per-partition ordering and avoids duplicate processing.
2. A topic has 4 partitions and a group has 6 consumers. What happens?
Parallelism is capped by the partition count, so with 4 partitions only 4 consumers get work and the other 2 remain idle.
3. Two separate consumer groups subscribe to the same topic. Each group receives?
Groups are independent; every group gets the entire stream and tracks its own offsets, enabling multiple apps to consume the same topic.
Flash Cards
What defines a consumer group? — Consumers sharing the same group.id that cooperatively read a topic's partitions.
Partition-to-consumer rule within a group? — Each partition is owned by exactly one consumer in the group.
What limits consumer parallelism? — The number of partitions; extra consumers beyond that stay idle.
What triggers a rebalance? — A consumer joining or leaving the group, causing partitions to be reassigned.