What Is a Kafka Consumer?
A Kafka consumer is a client application that reads records from one or more topic partitions by repeatedly calling poll() in a loop. Kafka is pull-based, not push-based: the broker never sends records to a consumer unprompted, the consumer asks for the next batch of records each time it calls poll(), specifying how long it is willing to wait if nothing new is available. This design lets each consumer read at its own pace and lets multiple independent applications read the same topic without interfering with each other.
Cricket analogy: Like a fielder at deep mid-wicket continuously watching for the ball rather than the ball being thrown directly at them; the consumer polls for new records instead of Kafka pushing them.
Offsets and Committing Progress
Each partition maintains a strictly increasing sequence number, the offset, for every record written to it, and a consumer tracks the offset of the next record it should read. Committing an offset (either automatically via enable.auto.commit=true on a timer, or manually via commitSync()/commitAsync()) records that progress in Kafka's internal __consumer_offsets topic, so if the consumer restarts or a partition is reassigned, it resumes from the last committed offset instead of re-reading everything or skipping data. Manual commits, especially after processing completes, give applications precise control over at-least-once versus at-most-once behavior.
Cricket analogy: Like a scorer marking exactly which ball of the over has been recorded so a rain delay resumption knows precisely where play stopped; offsets track the exact read position.
Deserialization and Consumer Configuration
Consumers need a key.deserializer and value.deserializer that exactly match the byte format the producer used, otherwise deserialization throws or silently produces garbage. Other important settings include fetch.min.bytes (how much data the broker should accumulate before responding), max.poll.records (how many records a single poll() call returns), and max.poll.interval.ms (how long a consumer can take processing a batch before the group coordinator considers it dead and triggers a rebalance).
Cricket analogy: Like a commentator needing to know which broadcast feed's language track to relay, similar to a consumer needing the correct deserializer for the incoming byte format.
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092");
props.put("group.id", "order-processing-service");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("enable.auto.commit", "false");
props.put("auto.offset.reset", "earliest");
props.put("max.poll.records", 500);
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("orders"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> record : records) {
process(record.key(), record.value());
}
consumer.commitSync();
}
}auto.offset.reset (earliest or latest) only takes effect when there is no committed offset for the consumer group on that partition, such as the very first run or after offsets have expired. It does NOT reset offsets on every restart, a common source of confusion when consumers appear to skip or re-read data unexpectedly.
- Kafka is pull-based: consumers call poll() in a loop rather than being pushed records.
- Offsets track a consumer group's exact read position per partition.
- Committing offsets (auto or manual) persists progress to the __consumer_offsets topic.
- Manual commits after processing give precise control over delivery semantics.
- key.deserializer/value.deserializer must match the producer's serializers exactly.
- auto.offset.reset only applies when no committed offset exists for the group.
- max.poll.interval.ms bounds processing time per batch before triggering a rebalance.
Practice what you learned
1. How does a Kafka consumer receive new records?
2. Where are committed consumer offsets stored?
3. When does auto.offset.reset actually take effect?
4. What happens if key.deserializer does not match the producer's key.serializer?
5. What does max.poll.interval.ms bound?
Was this page helpful?
You May Also Like
Kafka Producers Explained
How Kafka producer clients serialize, batch, and send records to topic partitions, and the configuration knobs that control durability and throughput.
Consumer Groups
How Kafka consumer groups distribute partitions across cooperating consumers to scale reads horizontally, and how rebalancing keeps that assignment correct.
Delivery Guarantees in Kafka
The at-most-once, at-least-once, and exactly-once delivery semantics Kafka supports, and the producer/consumer configuration choices that determine which one you get.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics