etcd Explained

Analogy categoryAnalogies

etcd is a distributed, consistent key-value store built on the Raft consensus algorithm, and in Kubernetes it is the single source of truth: every object—Pods, Secrets, ConfigMaps, CustomResources—is serialized as protobuf and stored under a flat keyspace like /registry/pods/default/my-pod. Only kube-apiserver talks to etcd directly, which keeps etcd's data model decoupled from Kubernetes semantics and lets etcd be reused as a general-purpose coordination store outside Kubernetes entirely, such as in CoreDNS or Rook.

🏏

Cricket analogy: The official ICC scorebook is the single authoritative record of every run, wicket, and over in a Test match, and other systems like broadcast graphics just read from it—much like etcd is Kubernetes's single source of truth that other tools like CoreDNS also rely on.

The Raft Consensus Algorithm

Raft elects a single leader among the cluster members through randomized election timeouts, and once elected, only the leader accepts writes, replicating each log entry to followers and considering it committed once a majority acknowledge it, at which point the leader applies it to its state machine and responds to the client. If the leader stops sending heartbeats—because of a crash or network partition—followers time out and start a new election, and a candidate must receive votes from a majority of the cluster before becoming the new leader, which is why etcd deployments always use an odd number of members.

🏏

Cricket analogy: When a captain gets injured mid-match, the vice-captain doesn't just self-appoint—the team and umpires need clear confirmation before decisions from the new captain are treated as official, mirroring Raft's election process requiring majority votes before a new leader is recognized.

Data Model: Keys, Revisions, and Watch

etcd's data model is a flat keyspace of arbitrary byte strings, but every write also increments a global revision number, allowing clients to read a consistent snapshot at any prior revision or watch for all changes since a given revision without missing events—this is exactly the mechanism the API server uses to serve consistent list-then-watch semantics to controllers. Older revisions accumulate until compaction removes them, and a Get at a compacted revision returns an error, so etcd needs to be compacted periodically and defragmented afterward to reclaim disk space fragmented by compaction.

🏏

Cricket analogy: Ball-by-ball commentary logs give every delivery a sequential number, letting a fan tune in mid-match and catch up from exactly where they left off, mirroring etcd's revision numbers letting a watcher resume from any point without missing events.

bash

# Take a consistent snapshot of the etcd data store
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%F).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify the snapshot's health before trusting it for recovery
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot-2026-07-10.db --write-out=table

# Defragment a member to reclaim space after heavy compaction
ETCDCTL_API=3 etcdctl defrag --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Defragmenting an etcd member blocks that member from serving requests while it runs, and it should be performed one member at a time with health checks in between — defragmenting all members simultaneously can drop the cluster below quorum and cause an outage.

Operating etcd in Production

Because etcd is on the critical path for every cluster write, operators size it with fast, low-latency SSD storage and keep network round-trips between members under etcd's default heartbeat interval, since disk fsync latency is etcd's most common performance bottleneck. Regular snapshots via etcdctl snapshot save are essential disaster-recovery insurance—restoring a cluster from a stale or corrupted etcd is often faster than trying to reconstruct state object by object—and production runbooks typically automate periodic snapshotting to object storage alongside monitoring etcd's db_size and wal_fsync_duration_seconds metrics for early warning of trouble.

🏏

Cricket analogy: A stadium's electronic scoreboard needs a low-latency feed to stay in sync with live play, or fans see stale scores, just as etcd needs low-latency SSDs or writes stall behind slow fsyncs.

The etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds metrics are the first place to look when etcd feels slow — sustained fsync latency above roughly 10ms on the 99th percentile is a strong signal to move etcd onto faster local SSDs rather than network-attached storage.

etcd is a distributed key-value store using Raft consensus as Kubernetes' single source of truth.
Only kube-apiserver communicates directly with etcd.
A write commits once a majority of etcd members acknowledge it.
Every write increments a global revision number, enabling consistent snapshot reads and resumable watches.
Compaction removes old revisions; defragmentation reclaims disk space afterward.
Disk fsync latency is etcd's most common performance bottleneck.
Regular snapshots via etcdctl snapshot save are essential for disaster recovery.