ZooKeeper
By the Apache Software Foundation
Apache ZooKeeper is an open-source distributed coordination service that provides a shared hierarchical namespace, similar to a filesystem, which distributed applications use to implement configuration management, naming, leader election,…
Definition
Apache ZooKeeper is an open-source distributed coordination service that provides a shared hierarchical namespace, similar to a filesystem, which distributed applications use to implement configuration management, naming, leader election, and distributed locking. It maintains strong consistency guarantees across a cluster of servers using a consensus protocol, giving distributed systems a reliable building block for coordination problems that are otherwise difficult to solve correctly from scratch.
Overview
Building distributed systems requires solving recurring coordination problems, such as electing a single leader among many nodes, managing shared configuration that all nodes must agree on, and implementing distributed locks, and getting these primitives right independently in every application is difficult and error-prone. ZooKeeper was created at Yahoo and later became an Apache Software Foundation project to provide these primitives as a shared, well-tested service that other distributed systems could depend on rather than reimplementing. ZooKeeper organizes data in a hierarchical structure of nodes called znodes, conceptually similar to a filesystem's directory tree, where each znode can hold a small amount of data and can be ephemeral, existing only as long as the client session that created it remains active, or persistent. A cluster, called an ensemble, of ZooKeeper servers replicates this data and uses the ZAB (ZooKeeper Atomic Broadcast) consensus protocol to keep all servers in agreement, with one server acting as a leader that orders writes and the rest as followers that replicate them; clients can connect to any server in the ensemble and see a consistent view of the data. Ephemeral znodes and watches, which notify clients when a znode changes, are the building blocks applications use to implement leader election and distributed locks. ZooKeeper is frequently compared to etcd and HashiCorp Consul, which solve similar coordination and configuration problems using the Raft consensus algorithm instead of ZAB. Kubernetes itself uses etcd rather than ZooKeeper as its cluster state store, and many newer distributed systems have chosen etcd or Consul over ZooKeeper, partly due to a simpler operational model and gRPC-based API, though ZooKeeper remains deeply embedded in older, widely deployed systems. Apache Kafka historically relied on ZooKeeper for broker coordination, controller election, and topic metadata storage, and this is the single most common reason many organizations run a ZooKeeper ensemble today, though newer Kafka versions have introduced KRaft mode to remove that dependency. Apache Hadoop, HBase, and Solr are other well-known systems that have used ZooKeeper for coordination. Operating a ZooKeeper ensemble reliably at scale requires careful attention to ensemble sizing, which must be an odd number of servers for quorum math to work, disk and network latency, since ZooKeeper's consensus protocol is latency-sensitive, and version upgrades. As newer coordination systems have matured and simplified some of these operational concerns, and as Kafka itself moves away from requiring it, ZooKeeper's footprint in new system designs has been shrinking even as it remains critical infrastructure in many existing production deployments.
Key Features
- Hierarchical znode namespace similar to a filesystem
- ZAB consensus protocol keeps an ensemble of servers strongly consistent
- Ephemeral znodes tied to client session lifetime enable leader election
- Watches notify clients when znode data or children change
- Used historically by Apache Kafka for broker and controller coordination
- Requires an odd-numbered ensemble size for correct quorum behavior