What is a Kafka broker and how does a Kafka cluster work?
Understand what a Kafka broker is, how a cluster distributes partitions, leader-follower replication, the controller and failover, with clear examples.
Expected Interview Answer
A Kafka broker is a single server in a Kafka cluster that stores partition data, serves producer writes and consumer reads, and participates in replication; a cluster is a set of brokers that together host all topics and coordinate through a controller to stay fault tolerant.
Each partition has one leader broker that handles all reads and writes and one or more follower brokers that replicate it. A controller broker (elected via ZooKeeper in older versions or KRaft in newer ones) tracks membership and reassigns leadership when a broker fails. Producers and consumers only need a bootstrap broker to discover the full cluster, then talk directly to the leader of each partition they use.
- Distributes topic partitions across many machines
- Replication gives fault tolerance and high availability
- Leader election lets the cluster survive broker failures
- Clients auto-discover the cluster from one bootstrap server
- Horizontal scaling by simply adding more brokers
AI Mentor Explanation
Think of a broker as one member of a scoring panel and the cluster as the full panel running a tournament. Each match is owned by a lead scorer while others keep backup copies of that sheet. If the lead scorer steps away, a backup is promoted instantly so scoring never stops, and a head scorer tracks who is covering what across every match.
Step-by-Step Explanation
Step 1
Brokers store partitions
Each broker holds a subset of the cluster's partitions on disk and serves the clients that read from or write to them.
Step 2
Leaders and followers
Every partition has one leader broker handling all traffic and follower brokers that replicate its data for redundancy.
Step 3
A controller coordinates
One broker acts as the controller, tracking membership and electing new partition leaders when a broker fails.
Step 4
Clients bootstrap the cluster
Producers and consumers connect to a bootstrap broker, fetch metadata, then talk directly to each partition's leader.
Step 5
Failover keeps it running
If a leader broker dies, an in-sync follower is promoted so writes and reads continue with minimal disruption.
What Interviewer Expects
- Broker as one server storing partitions and serving clients
- Leader/follower replication model per partition
- Role of the controller and leader election
- How ZooKeeper or KRaft coordinates the cluster
- How clients discover brokers from a bootstrap server
Common Mistakes
- Thinking a broker stores an entire topic rather than partitions
- Confusing the controller with a single point of failure
- Believing all brokers serve every partition's reads and writes
- Ignoring the in-sync replica set when discussing failover
- Assuming clients must know every broker address up front
Best Answer (HR Friendly)
“A Kafka broker is one server in a cluster that stores data and answers requests, and a cluster is a team of these servers that share the work and back each other up. If one server fails, another takes over its data automatically, so the system keeps running.”
Code Example
# List the brokers currently registered in the cluster
kafka-broker-api-versions.sh --bootstrap-server localhost:9092 | grep id
# Show which broker leads each partition and the in-sync replicas
kafka-topics.sh --describe \
--topic orders \
--bootstrap-server localhost:9092
# Output lists Leader, Replicas and Isr per partitionFollow-up Questions
- What is the role of the Kafka controller?
- How does Kafka elect a new partition leader when a broker fails?
- What is the difference between ZooKeeper mode and KRaft mode?
- What is an in-sync replica (ISR) set?
- How does a client discover all brokers from a bootstrap server?
MCQ Practice
1. What does a single Kafka broker primarily store?
A broker stores a subset of the cluster's partitions and serves reads and writes for the ones it leads.
2. Which broker handles all reads and writes for a given partition?
Each partition has one leader that handles all client traffic; followers only replicate its data.
3. What is the controller broker responsible for?
The controller tracks broker membership and elects new partition leaders when brokers join or fail.
Flash Cards
What is a Kafka broker? — A single server in a cluster that stores partitions, serves clients, and participates in replication.
What is a partition leader? — The one broker that handles all reads and writes for a partition while followers replicate it.
What does the controller do? — Tracks cluster membership and elects new partition leaders when brokers fail.
How do clients find the cluster? — They connect to a bootstrap broker, fetch metadata, then talk directly to each partition leader.