What is Apache Kafka and what problems does it solve?
Learn what Apache Kafka is, how its producer-topic-consumer model works, and the problems it solves: decoupling, durability, replay, and scalable streaming.
Expected Interview Answer
Apache Kafka is a distributed, durable event-streaming platform that lets systems publish, store, and subscribe to continuous streams of records at high throughput, decoupling the producers of data from the consumers that process it.
Kafka solves the problem of many systems needing the same data in real time without wiring every producer directly to every consumer. Producers append records to append-only, partitioned logs called topics; brokers persist those records durably and replicate them across the cluster; and any number of consumers read at their own pace, replaying history if needed. This turns brittle point-to-point integrations into a single, scalable backbone for pipelines, event-driven microservices, metrics, and log aggregation.
- Decouples producers from consumers so services evolve independently
- Handles very high throughput with horizontal scaling
- Durable, replayable storage so consumers can re-read past events
- Fault tolerant through partition replication across brokers
- Delivers data to many consumers in real time from one source
AI Mentor Explanation
Kafka is like the official ball-by-ball commentary feed at a stadium: every delivery is recorded in order onto one continuous log, and the broadcasters, statisticians, and app developers all tap the same feed without the umpire calling each of them personally. New subscribers can rewind to over one and replay every ball, and if one screen fails the feed keeps flowing for everyone else.
Step-by-Step Explanation
Step 1
Producers publish events
Applications write records to a named topic; each record has an optional key and a value.
Step 2
Brokers store durably
The Kafka cluster appends records to partitioned, on-disk logs and replicates each partition across brokers.
Step 3
Data is retained
Records stay for a configured retention period, so they can be re-read rather than deleted on consumption.
Step 4
Consumers subscribe
Independent consumer groups read the topic at their own pace, each tracking its own offset.
Step 5
Scale out
Adding partitions and brokers increases throughput and lets more consumers process in parallel.
What Interviewer Expects
- A clear definition: distributed event-streaming and durable log
- The producer, topic, and consumer model
- Why decoupling matters versus point-to-point integration
- At least one real use case (log aggregation, metrics, event-driven microservices)
- Awareness of durability and replay, not just messaging
Common Mistakes
- Calling Kafka just a message queue with nothing more to say
- Ignoring durability and replay, since Kafka stores events, not just forwards them
- Confusing Kafka with a database for random-access queries
- Not mentioning partitions or horizontal scaling
- Assuming messages are deleted the moment a consumer reads them
Best Answer (HR Friendly)
“Apache Kafka is a system that carries a continuous stream of events between applications. Instead of connecting every app to every other app, everyone reads and writes from one shared, durable stream, which keeps large systems fast, reliable, and easy to extend.”
Code Example
// Producer: publish an order event
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", StringSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
producer.send(new ProducerRecord<>("orders", "order-42", "{\"total\":99}"));
}
// Consumer: read the same events independently
consumer.subscribe(List.of("orders"));
for (ConsumerRecord<String, String> record : consumer.poll(Duration.ofMillis(100))) {
System.out.println(record.key() + " -> " + record.value());
}Follow-up Questions
- How does Kafka guarantee durability of published events?
- What is a consumer group and how does it enable parallelism?
- When would you choose Kafka over a traditional message broker?
- What are some common real-world use cases for Kafka?
- How does Kafka scale to handle millions of events per second?
MCQ Practice
1. Which phrase best describes Apache Kafka?
Kafka is a distributed, durable platform for publishing, storing, and subscribing to streams of events.
2. What primary problem does Kafka address?
Kafka replaces brittle point-to-point integrations with one durable stream that many consumers can read independently.
3. After a consumer reads a Kafka record, the record is:
Kafka retains records for a configured time, so multiple consumers can read and replay them.
Flash Cards
What is Apache Kafka in one line? — A distributed, durable event-streaming platform for publishing, storing, and subscribing to streams of records.
What core problem does Kafka solve? — It decouples producers from consumers so many systems can share the same real-time data at scale.
Where does Kafka store events? — In append-only, partitioned logs called topics, replicated durably across brokers.
Can consumers re-read old Kafka events? — Yes. Records are retained for a configured period, so consumers can replay history by resetting offsets.