Cluster reliability — multi-AZ architecture, Velero backup and chaos engineering

A Kubernetes cluster that is correctly configured, secured, and observable can still experience reliability failures from causes that are entirely outside the application and infrastructure stack: an AWS availability zone failure that takes all nodes in one AZ offline, an accidental `kubectl delete namespace production` that destroys all resources in the production namespace, or a cascading failure triggered by a network partition that the system had never experienced in testing. Production reliability requires architecture beyond correct configuration — it requires multi-AZ topology that distributes workloads across independent failure domains, backup and restore capabilities that recover from catastrophic data loss faster than building from scratch, and chaos engineering practices that proactively inject failures in controlled conditions to discover reliability gaps before they manifest as production outages. These three practices together implement the reliability discipline that distinguishes a cluster that works from a cluster that is resilient.

Analogy🏏Cricket

🏏 Think of it like cricket: The Pod-ReplicaSet-Deployment hierarchy maps precisely onto the three levels of IPL franchise team management. A Pod is a single player on the field at a given moment — the smallest unit of participation, carrying its own identity and fulfilling a specific role in the current game. A ReplicaSet is the franchise's match-day playing XI contract — it specifies that exactly eleven players matching a specific profile must always be on the field; if one is injured and leaves, the team management immediately sends a substitute of the same profile to restore the count. A Deployment is the franchise's season-long team strategy — it manages how the playing XI evolves between matches: when a new batting approach is adopted, the Deployment replaces the old XI with the new one in a controlled rolling substitution rather than swapping all eleven players simultaneously and disrupting team cohesion. Just as the franchise director does not manage individual players directly — the playing XI contract (ReplicaSet) handles the count and the season strategy (Deployment) handles the transitions — you never manage Pods directly in production; the Deployment manages the transition and the ReplicaSet maintains the count. This reveals why the three-level hierarchy exists rather than one omnibus 'workload' object: each level solves one specific problem, and composing three focused abstractions produces better separation of concerns than one object that conflates scheduling, scaling, and update management.