What Is Apache Kafka?

Analogy categoryAnalogies

Apache Kafka is a distributed event streaming platform originally built at LinkedIn in 2010-2011 by Jay Kreps, Neha Narkhede, and Jun Rao to move activity data (page views, clicks, logs) at massive scale, then open-sourced through the Apache Software Foundation in 2011. At its core, Kafka lets producer applications publish records to named topics, stores those records durably on disk in the order they arrive, and lets any number of consumer applications read them independently, at their own pace, making it simultaneously a messaging system, a storage layer, and a foundation for stream processing.

🏏

Cricket analogy: Think of Kafka's topic like the official ball-by-ball commentary feed of an IPL match between Mumbai Indians and Chennai Super Kings: it's written once by the scorer and then broadcast to millions of listeners on radio, TV, and apps simultaneously, each consuming it independently.

Why Kafka Exists: The Problem It Solves

Before systems like Kafka existed, organizations typically wired data flows point-to-point: each source system built a custom integration with each destination system, producing an O(N x M) tangle of brittle connections that broke whenever a schema changed or a new consumer was added. Kafka inverts this by acting as a central nervous system: producers write once to a topic, and any number of current or future consumers can read that same data independently, without the producer ever needing to know who is listening or how many downstream systems exist.

🏏

Cricket analogy: Before a central scoring feed existed, every broadcaster had to send a runner to relay scores from the scorer's table individually; a shared scoreboard feed (Kafka's role) means Star Sports, Doordarshan, and a stadium's giant screen all read one source instead of N separate wires.

Core Concepts at a Glance

A handful of vocabulary terms recur throughout Kafka: a producer is a client application that writes records to a topic; a topic is a named, append-only log that records are published to; a partition is a topic's unit of parallelism and ordering, and a broker is a Kafka server that stores partitions and serves reads/writes; a consumer reads records from partitions, typically as part of a consumer group that divides the partitions among its members for parallel processing. Records are not deleted on read the way a traditional queue's messages are; instead they persist for a configurable retention period (time-based or size-based), which is what lets multiple independent consumer groups replay the same data at different times.

🏏

Cricket analogy: A topic is like a stadium's over-by-over ledger for a Test match: the scorer (producer) fills it in ball by ball, the ledger itself is split across several scorebooks (partitions) for different innings, and commentators (consumers) can flip back to review any over long after it happened.

bash

# Start a local Kafka broker (KRaft mode, no ZooKeeper needed in modern Kafka)
bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/kraft/server.properties
bin/kafka-server-start.sh config/kraft/server.properties

# Create a topic with 3 partitions and replication factor 1 (single broker)
bin/kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092 \
  --partitions 3 --replication-factor 1

# Produce a couple of records from the console
echo "order-101:{\"item\":\"laptop\",\"qty\":1}" | bin/kafka-console-producer.sh \
  --topic orders --bootstrap-server localhost:9092 --property parse.key=true --property key.separator=:

# Consume from the beginning of the topic
bin/kafka-console-consumer.sh --topic orders --bootstrap-server localhost:9092 \
  --from-beginning --property print.key=true

Common Use Cases

In production, Kafka is most often used for log and metrics aggregation (collecting application logs or system metrics from thousands of hosts into a central pipeline), event-driven microservices (services publish domain events like OrderPlaced or PaymentFailed instead of calling each other synchronously), change data capture (tools like Debezium stream every row-level database change into Kafka topics), and real-time stream processing with libraries like Kafka Streams or ksqlDB to compute rolling aggregates, joins, and windowed analytics directly on the event stream. Companies including LinkedIn, Netflix, Uber, and Airbnb run Kafka clusters processing trillions of messages per day to power everything from fraud detection to recommendation pipelines.

🏏

Cricket analogy: Just as the BCCI aggregates ball-by-ball data from every IPL stadium into one central feed to power live win-probability graphics, companies aggregate application logs from thousands of servers into Kafka to power real-time dashboards and alerting.

Unlike a traditional message queue (e.g., RabbitMQ) where a message is typically removed once consumed, Kafka retains records for a configurable retention period (time-based, e.g. 7 days, or size-based) regardless of whether they've been read. This log-based storage model is what makes it possible for multiple independent consumer groups to read the same topic at different speeds, and for new consumers to replay historical data.

Kafka is not a drop-in replacement for a relational database or a general-purpose OLTP store. While compacted topics and ksqlDB tables can serve some lookup patterns, Kafka lacks ad-hoc query flexibility, secondary indexes, and transactional guarantees across arbitrary reads and writes that a database provides. Treating a Kafka topic as your primary source-of-truth datastore without careful design is a common and costly architectural mistake.

Kafka is a distributed event streaming platform built at LinkedIn and open-sourced in 2011.
It decouples producers from consumers, solving the N-by-M point-to-point integration problem.
Core building blocks are producers, topics, partitions, brokers, and consumers/consumer groups.
Records persist for a configurable retention period rather than being deleted on read.
Common use cases include log aggregation, event-driven microservices, CDC, and stream processing.
Kafka is a messaging, storage, and stream-processing platform combined, not a traditional queue.
It is not a general-purpose database replacement despite supporting compacted, table-like topics.