What Is Apache Kafka and How Does It Work?
Understand how Apache Kafka works — topics, partitions, offsets, and consumer groups — explained for DevOps interview preparation.
Expected Interview Answer
Apache Kafka is a distributed, durable event-streaming platform that stores records as an ordered, append-only log split into partitions, allowing many independent consumers to read the same stream of events at their own pace without removing messages from it.
Producers write records to a named topic, which Kafka splits across multiple partitions for parallelism, and each record is appended in order within its partition and assigned a monotonically increasing offset. Unlike a traditional queue, Kafka does not delete a message once it is read; instead it retains records for a configured retention period, so multiple independent consumer groups can each replay or read the same topic at their own offset. Consumers within a consumer group split the partitions among themselves for parallel processing, while consumers in different groups each get their own independent read position. Kafka brokers replicate each partition across multiple nodes for durability, and a partition leader handles all reads and writes for that partition while followers stay in sync, so the cluster tolerates broker failures without losing committed data.
- High-throughput, ordered event streaming with horizontal scalability
- Durable retention lets multiple consumers replay the same events independently
- Partitioning enables parallel processing while preserving per-key order
- Replication provides fault tolerance across broker failures
AI Mentor Explanation
Kafka is like a full match scorecard broadcast that keeps a permanent ball-by-ball log rather than a live commentary that vanishes once spoken. Any fan can tune into the feed and read from over one while another fan reads from over forty, because the scorecard log is never erased after being read. Different broadcasters (consumer groups) can each track their own position through the same match log independently. The stadium's multiple scoreboard screens (partitions) each track separate innings segments so many scorers can update in parallel while still keeping each segment strictly in order.
Step-by-Step Explanation
Step 1
Producer writes to a topic
A record is appended to a partition within the topic and assigned an increasing offset.
Step 2
Broker replicates and persists
The partition leader replicates the write to followers and retains it on disk for the configured retention window.
Step 3
Consumer groups read independently
Each consumer group tracks its own offset per partition, so multiple groups can read the same data independently.
Step 4
Partitions enable parallelism
Consumers within a group split partitions among themselves to process the topic in parallel while preserving per-partition order.
What Interviewer Expects
- Understanding that Kafka retains records instead of deleting them on read
- Knowledge of topics, partitions, offsets, and consumer groups
- Awareness of replication and leader/follower roles for fault tolerance
- Ability to contrast Kafka with a traditional point-to-point message queue
Common Mistakes
- Calling Kafka just another message queue without mentioning its log-based retention
- Forgetting that ordering is only guaranteed within a single partition
- Confusing consumer group semantics with simple pub/sub fan-out
- Ignoring partition key choice, which determines both ordering and load balance
Best Answer (HR Friendly)
“Kafka is a durable, high-throughput event log — instead of a message disappearing once it is read like a traditional queue, Kafka keeps a full history that different teams can independently replay. We use it when several systems need to react to the same stream of events, at different speeds, without stepping on each other.”
Code Example
# Create a topic with 3 partitions and replication factor 2
kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server broker:9092
# Produce a record
echo “order-1234” | kafka-console-producer.sh --topic orders --bootstrap-server broker:9092
# Consume as part of a named group, tracking its own offset
kafka-console-consumer.sh --topic orders --group billing-service --bootstrap-server broker:9092 --from-beginningFollow-up Questions
- How does Kafka guarantee ordering, and what is the scope of that guarantee?
- What determines which partition a given record is written to?
- How does a consumer group rebalance when a consumer instance joins or leaves?
- How would you design a partition key to avoid a hot partition?
MCQ Practice
1. What happens to a Kafka record after it has been read by a consumer?
Kafka retains records for a configured retention window regardless of whether they have been consumed, allowing replay.
2. Within what scope does Kafka guarantee strict message ordering?
Kafka guarantees order only within a partition; records across different partitions of a topic have no relative ordering guarantee.
3. What allows two different applications to independently consume the same Kafka topic at their own pace?
Each consumer group maintains its own offsets, so distinct groups can read the same topic independently without interfering.
Flash Cards
What is Kafka fundamentally? — A distributed, durable, append-only log for event streaming.
What is a Kafka partition? — An ordered, appendable segment of a topic that enables parallelism.
What is a consumer group? — A set of consumers sharing partitions, tracking one independent offset per partition.
How does Kafka survive broker failure? — Partitions are replicated across brokers with a leader and in-sync followers.