Apache ZooKeeper
Apache ZooKeeper is an open-source distributed coordination service that provides a hierarchical key-value store, consensus, and primitives such as locks and leader election for building reliable distributed systems.
Definition
Apache ZooKeeper is an open-source distributed coordination service that provides a hierarchical key-value store, consensus, and primitives such as locks and leader election for building reliable distributed systems.
Overview
ZooKeeper was created at Yahoo to centralize the coordination logic that many distributed systems need to implement independently, such as configuration management, naming, and synchronization. It organizes data as a tree of nodes called znodes, similar to a filesystem, and clients can read, write, and watch these nodes for changes, which lets applications react to configuration or membership updates in near real time. Under the hood, ZooKeeper achieves consistency using the Zab atomic broadcast protocol, where a leader node orders all writes and replicates them to followers, requiring a majority (quorum) of the ensemble to acknowledge each write. This gives ZooKeeper strong consistency guarantees similar in spirit to those provided by etcd, though its API and data model differ. For years, ZooKeeper was the default coordination layer for the big data ecosystem, underpinning systems such as Apache HBase, Apache Kafka (in its earlier architecture), and Apache Hadoop for tasks like leader election and distributed locking. While some newer projects have adopted etcd or built their own consensus layers, ZooKeeper remains deeply embedded in many production big data and distributed systems deployments.
Key Features
- Hierarchical, filesystem-like data model (znodes)
- Strong consistency via the Zab atomic broadcast protocol
- Watches for real-time notification of data or membership changes
- Ephemeral nodes for automatic cleanup of failed clients
- Distributed locks, leader election, and barrier primitives
- Battle-tested coordination layer for the Hadoop/Kafka ecosystem
- Client libraries available for most major programming languages