Why Kafka Needed a Metadata Layer
For most of Kafka's history, cluster metadata — the list of topics, partitions, replica assignments, ACLs, and which broker is currently the controller — was stored in an external Apache ZooKeeper ensemble rather than inside Kafka itself. Brokers watched ZooKeeper znodes for changes and used ZooKeeper's ephemeral nodes and session mechanism for controller election and broker liveness detection, which meant every Kafka cluster required operators to separately deploy, tune, and secure a completely different distributed system with its own quorum, ports, and failure modes.
Cricket analogy: Historically, a cricket board outsourced ticketing entirely to a third-party vendor with its own servers and staff, meaning every stadium had to integrate with a separate outside system just to know who's supposed to be seated where, like Kafka depending on an external ZooKeeper ensemble for metadata.
How KRaft Replaces ZooKeeper
KRaft (Kafka Raft), introduced via KIP-500 and made the default in Kafka 3.x/4.x, removes the ZooKeeper dependency entirely by storing metadata as an event-sourced log replicated using the Raft consensus protocol among a set of controller nodes, written to an internal topic called __cluster_metadata. Instead of brokers polling and watching znodes, brokers now consume this metadata log directly, much like consumers reading any other Kafka topic, which unifies Kafka's operational model around a single technology (Kafka's own log replication) instead of two.
Cricket analogy: Instead of outsourcing ticketing, a modern cricket board built ticketing directly into its own existing stadium management platform, so there's only one system to run, mirroring KRaft folding metadata into Kafka's own log replication instead of a separate ZooKeeper system.
Migration and Deployment Modes
KRaft nodes run in one of two modes, set via process.roles: controller-only nodes that participate solely in the Raft quorum and metadata management, or combined broker+controller nodes (process.roles=broker,controller) suitable for smaller clusters. Migrating an existing cluster from ZooKeeper to KRaft is a supported, staged process (KIP-866) involving provisioning a KRaft controller quorum, enabling ZK migration mode so brokers dual-write metadata, then finally cutting brokers over to KRaft-only and decommissioning ZooKeeper — a mistake in following the staged order risks metadata inconsistency.
Cricket analogy: A small local cricket club has one person who both umpires and manages the scorebook, while a national board has dedicated umpires separate from dedicated scorers, mirroring KRaft's combined mode for small clusters versus dedicated controller nodes for larger ones.
# server.properties for a combined broker+controller KRaft node
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@host1:9093,2@host2:9093,3@host3:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
controller.listener.names=CONTROLLEROperational and Scalability Differences
Operationally, KRaft removes an entire class of failure modes and tuning knobs tied to running ZooKeeper (session timeouts, jute.maxbuffer sizing, separate TLS/ACL configuration) and dramatically improves controller failover time and metadata scalability, since Raft-based leader election typically completes in a couple of seconds versus ZooKeeper session-timeout-bound elections that could take tens of seconds, and clusters can now support far more partitions because metadata is no longer bottlenecked by ZooKeeper's in-memory znode tree and single-writer model.
Cricket analogy: Merging ticketing into the stadium's own system means fewer outside vendors to coordinate with, and if there's an outage, the board's own on-call team can fix it directly instead of waiting on a vendor, like KRaft removing an entire external ZooKeeper failure mode.
ZooKeeper mode is deprecated and removed starting in Kafka 4.0 — any cluster still running in ZK mode must plan a KIP-866 migration to KRaft before upgrading, since new major versions no longer ship ZooKeeper support at all.
- Historically Kafka stored cluster metadata in an external ZooKeeper ensemble.
- KRaft (KIP-500) removes the ZooKeeper dependency by replicating metadata via Raft into an internal __cluster_metadata topic.
- KRaft nodes run as controller-only or combined broker+controller, set via process.roles.
- Migrating from ZooKeeper to KRaft is a staged, supported process defined by KIP-866.
- KRaft offers faster controller failover than ZooKeeper session-timeout-bound elections.
- ZooKeeper mode is removed starting in Kafka 4.0, requiring migration before upgrading.
Practice what you learned
1. What did Kafka historically use ZooKeeper for?
2. What internal topic does KRaft use to store its replicated metadata log?
3. What config determines whether a KRaft node is a dedicated controller or combined broker+controller?
4. What KIP defines the staged migration process from ZooKeeper to KRaft?
5. Starting with which Kafka version is ZooKeeper mode removed entirely?
Was this page helpful?
You May Also Like
The Kafka Controller
The broker role responsible for cluster-wide coordination: leader election, metadata propagation, and admin operations.
Kafka Replication and Leaders
How Kafka replicates partitions across brokers and elects leaders to guarantee durability and availability.
Retention Policies in Kafka
How Kafka's time- and size-based retention deletes old log segments, and how it interacts with consumer lag.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics