What is etcd and Why Does Kubernetes Depend on It?
Learn what etcd is, why Kubernetes stores all cluster state there, and how Raft consensus and quorum keep it consistent.
Expected Interview Answer
etcd is a distributed, strongly consistent key-value store that serves as the single source of truth for all Kubernetes cluster state — every object, from Pods to Secrets to the cluster's current configuration, is persisted there, and only the API server talks to it directly.
etcd uses the Raft consensus algorithm to replicate data across an odd number of member nodes (typically 3 or 5), electing a leader that handles writes and replicates them to followers, so a majority of nodes must agree before a write is committed — this gives etcd strong consistency and tolerance of a minority of node failures without data loss or split-brain. The Kubernetes API server is etcd's only direct client: controllers, the scheduler, and kubelets never read or write etcd themselves, they always go through the API server, which enforces validation, admission control, and RBAC before any change is persisted. This centralization is also why etcd health and latency directly gate overall cluster responsiveness — a slow or unhealthy etcd quorum makes the entire API server sluggish, since every kubectl apply, controller reconcile, and scheduling decision ultimately reads or writes through it. Because etcd holds Secrets and the entire cluster's state, its data directory must be encrypted at rest and backed up regularly, since losing etcd effectively means losing the cluster.
- Provides a strongly consistent, single source of truth for cluster state
- Raft consensus tolerates minority node failure without data loss
- Centralizing writes through the API server enables uniform validation and RBAC
- Watch-based reads let controllers react to state changes efficiently
AI Mentor Explanation
etcd is like the official, single scorebook that a panel of odd-numbered scorers keeps in sync during a match — a run is only recorded as official once a majority of the scorers agree it happened, which prevents any one scorer's mistake from corrupting the record. Only the match referee is allowed to actually write into that scorebook; players and coaches never touch it directly, they report to the referee, who validates the claim first. If the scorebook panel loses too many members at once and cannot reach a majority, no new runs can be officially recorded until enough scorers are restored. Because that scorebook is the only authoritative record of the match, losing it entirely means losing the ability to know the true state of the game.
Step-by-Step Explanation
Step 1
API server receives a write
kubectl apply or a controller sends a create/update request to the API server, which runs admission control and RBAC checks.
Step 2
API server writes to etcd
Only the API server talks to etcd directly, persisting the validated object as a key-value entry.
Step 3
Raft replicates the write
etcd's leader replicates the write to follower nodes; the write commits only once a majority (quorum) acknowledges it.
Step 4
Controllers watch for changes
Controllers and the scheduler watch the API server (which streams from etcd) and reconcile the cluster to match the new state.
What Interviewer Expects
- Understanding etcd as the single source of truth, storing all cluster state as key-value pairs
- Knowledge that only the API server talks to etcd directly, never controllers or kubelets
- Awareness of Raft consensus, quorum, and why an odd number of etcd members is used
- Ability to explain the operational impact: etcd health directly gates cluster performance and requires backups/encryption
Common Mistakes
- Saying controllers or kubelets read/write etcd directly instead of going through the API server
- Not knowing why an odd number of etcd members (e.g. 3 or 5) is required for quorum
- Forgetting that losing etcd (with no backup) means losing the entire cluster's state
- Confusing etcd with a general-purpose application database rather than Kubernetes' internal state store
Best Answer (HR Friendly)
“etcd is the database that holds the entire state of a Kubernetes cluster — every object and configuration lives there. Only the API server is allowed to talk to it directly, and etcd uses a consensus protocol across multiple nodes so a write is only confirmed once a majority agree, which keeps the cluster's source of truth consistent and resilient, which is why we always back it up and monitor its health closely.”
Code Example
# Check etcd member list and health (run from a control-plane node)
ETCDCTL_API=3 etcdctl member list \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
# Take a consistent snapshot backup of etcd
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.keyFollow-up Questions
- What happens to the cluster if etcd loses quorum?
- Why does Kubernetes typically run etcd with an odd number of members like 3 or 5?
- How would you restore a cluster from an etcd snapshot backup?
- Why should etcd's data directory be encrypted at rest?
MCQ Practice
1. Which Kubernetes component is the only direct client of etcd?
Only the API server reads from and writes to etcd directly; all other components go through the API server.
2. Why does an etcd cluster typically use an odd number of members?
An odd number of members avoids ties and ensures a clear majority quorum can always be determined during leader election and writes.
3. What happens to write operations if the etcd cluster loses quorum?
Without a quorum, Raft cannot commit new writes, so the API server cannot persist changes until quorum is restored.
Flash Cards
What is etcd in Kubernetes? — The distributed, strongly consistent key-value store holding all cluster state.
Who talks to etcd directly? — Only the API server — controllers and kubelets never access it directly.
What consensus algorithm does etcd use? — Raft, requiring a majority (quorum) to commit writes.
Why back up etcd? — Losing etcd without a backup means losing the entire cluster's state.