What is the role of ZooKeeper in Kafka and what is KRaft mode?
Learn what ZooKeeper did for Kafka and how KRaft mode replaces it with a Raft controller quorum for simpler ops, faster failover and bigger scale.
Expected Interview Answer
Historically ZooKeeper stored Kafka's cluster metadata (broker registration, topic and partition configs, controller election, and ACLs), while KRaft is Kafka's newer mode that removes ZooKeeper by managing that metadata internally through a Raft-based quorum of controllers.
In the ZooKeeper architecture, brokers depend on an external ZooKeeper ensemble for coordination, and one broker acts as the controller elected via ZooKeeper. KRaft (Kafka Raft) instead stores metadata as an event log replicated by a dedicated controller quorum using the Raft consensus protocol, so Kafka becomes a single self-managed system. KRaft simplifies deployment, scales to far more partitions, speeds up controller failover and metadata propagation, and is the default for new clusters, with ZooKeeper deprecated and removed in Kafka 4.0.
- Removes a separate system to deploy and operate
- Scales to millions of partitions with faster metadata handling
- Much quicker controller failover and recovery
- Single security and configuration model
- Metadata stored as a replicated, auditable event log
AI Mentor Explanation
ZooKeeper is like relying on an external match referee's office to keep the official roster, toss result, and umpire assignments, with the players constantly phoning that office to stay coordinated. KRaft is like the teams electing a captains' council on the field that keeps the authoritative record among themselves via agreed voting, removing the outside office entirely. The game becomes self-governing, with decisions logged and replicated among the council rather than held by a separate body.
Step-by-Step Explanation
Step 1
Metadata needs coordination
A Kafka cluster must agree on brokers, topics, partition leaders, and configs; that shared state needs a consistent store.
Step 2
ZooKeeper's historical role
An external ZooKeeper ensemble held that metadata and elected the single controller broker for the cluster.
Step 3
Controller managed the rest
The elected controller pushed leadership and metadata changes to other brokers, with ZooKeeper as the source of truth.
Step 4
KRaft internalizes metadata
KRaft replaces ZooKeeper with a controller quorum that stores metadata as a Raft-replicated event log inside Kafka.
Step 5
Faster, simpler operations
KRaft speeds failover and metadata propagation, scales to more partitions, and unifies deployment into one system.
What Interviewer Expects
- What metadata ZooKeeper stored and why it was needed
- The concept of the controller and its election
- That KRaft uses Raft consensus with a controller quorum
- Concrete benefits: simpler ops, scale, faster failover
- Awareness that ZooKeeper is deprecated and removed in Kafka 4.0
Common Mistakes
- Thinking ZooKeeper stored the actual message data
- Believing brokers talk to ZooKeeper for every produce or consume
- Not knowing KRaft exists or that it replaces ZooKeeper
- Confusing the controller with a producer or consumer
- Assuming KRaft still needs an external quorum service
Best Answer (HR Friendly)
“ZooKeeper used to be a separate helper system that kept track of Kafka's cluster information, like which servers exist and who is in charge. KRaft is Kafka's newer setup that does all of that itself, so you no longer need to run and manage a second system alongside Kafka.”
Code Example
# Generate a unique cluster ID
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
# Format storage for the combined broker+controller node
bin/kafka-storage.sh format \
-t "$KAFKA_CLUSTER_ID" \
-c config/kraft/server.properties
# Start the broker in KRaft mode
bin/kafka-server-start.sh config/kraft/server.properties
# server.properties key settings:
# process.roles=broker,controller
# controller.quorum.voters=1@localhost:9093Follow-up Questions
- Why was ZooKeeper a scaling bottleneck for Kafka?
- How does the Raft consensus protocol work in KRaft?
- What is the controller quorum in KRaft mode?
- How do you migrate an existing cluster from ZooKeeper to KRaft?
- In which Kafka version was ZooKeeper support removed?
MCQ Practice
1. In the legacy architecture, what did ZooKeeper store for Kafka?
ZooKeeper held metadata such as broker registration, topic configs, ACLs, and coordinated controller election, not the message data itself.
2. KRaft mode manages metadata using?
KRaft stores metadata as a Raft-replicated event log managed by a dedicated controller quorum, removing the need for ZooKeeper.
3. A key benefit of KRaft over ZooKeeper is?
KRaft scales to many more partitions with faster controller failover and metadata propagation, while simplifying deployment to one system.
Flash Cards
What did ZooKeeper do for Kafka? — Stored cluster metadata and handled controller election and coordination.
What is KRaft? — Kafka's ZooKeeper-free mode that manages metadata via a Raft controller quorum.
Where does KRaft keep metadata? — As a Raft-replicated event log inside a dedicated controller quorum.
When was ZooKeeper removed? — It is deprecated and removed in Kafka 4.0 in favor of KRaft.