A Systematic Diagnosis Workflow
Effective Kubernetes troubleshooting follows a layered workflow: start with kubectl get pods -o wide to see pod phase and node placement, then kubectl describe pod for the Events section (which shows scheduling failures, image pull errors, and probe failures in chronological order), then kubectl logs (and kubectl logs --previous for a crashed container's last output before restart). Only after exhausting the pod-level evidence should you move up to node conditions (kubectl describe node), or down into the container with kubectl exec or an ephemeral debug container via kubectl debug. Jumping straight to guesswork like restarting the deployment skips the Events section, which is often where the actual root cause — a failed image pull, an unschedulable resource request, or a failing readiness probe — is spelled out explicitly.
Cricket analogy: It's like a third umpire working through evidence layers systematically: first the on-field replay angle, then Hawk-Eye trajectory, then snickometer audio — skipping straight to a guess without checking the replay is how wrong decisions happen.
Common Failure Patterns
CrashLoopBackOff means the container starts, exits, and Kubernetes keeps restarting it with exponential backoff — the fix lives in kubectl logs --previous, not in restarting the deployment. ImagePullBackOff usually means a wrong image tag, a private registry needing an imagePullSecret, or a rate-limited public registry. Pending pods that never schedule are almost always a resource request the scheduler can't satisfy (check kubectl describe pod's Events for 'Insufficient cpu/memory' or a taint the pod lacks toleration for), while OOMKilled (visible as exit code 137 with reason OOMKilled in kubectl describe pod) means the container exceeded its memory limit and was killed by the kernel's cgroup OOM killer, distinct from a liveness-probe-triggered restart which shows a different reason entirely.
Cricket analogy: It's like a batter given out hit-wicket repeatedly at the crease — the umpire (Kubernetes) keeps sending them back in, but replaying the same delivery over and over won't fix a technical flaw that needs addressing in the nets, not on the field.
Debugging Networking and Ephemeral Containers
# Inspect why a pod is stuck Pending
kubectl describe pod checkout-7f9d8c-abcde -n payments | tail -20
# Get the last container's exit reason before a restart
kubectl logs checkout-7f9d8c-abcde -n payments --previous
# Attach an ephemeral debug container (no shell in the base image needed)
kubectl debug -it checkout-7f9d8c-abcde -n payments \
--image=busybox:1.36 --target=checkout -- sh
# Test DNS and service reachability from inside the cluster
kubectl run tmp-shell --rm -it --image=nicolaka/netshoot -- \
sh -c "nslookup checkout-service.payments.svc.cluster.local && curl -sv checkout-service.payments:8080/health"kubectl debug's ephemeral containers (stable since Kubernetes 1.25) let you attach a fully-featured debugging image to a running pod's network and process namespace without modifying the pod spec or requiring a shell to already exist in the distroless production image — essential for minimal images that ship no shell at all.
- Follow the evidence layer by layer: pod status, Events, logs, node conditions, then exec/debug.
- CrashLoopBackOff root causes live in kubectl logs --previous, not in restarting the deployment.
- Pending pods usually mean unsatisfiable resource requests or missing taint tolerations — check Events.
- OOMKilled (exit 137) means the cgroup memory limit was exceeded, distinct from probe-triggered restarts.
- ImagePullBackOff is almost always a bad tag, missing imagePullSecret, or registry rate limiting.
- kubectl debug attaches an ephemeral debug container without modifying the pod or needing an in-image shell.
- netshoot-style debug pods are the standard way to test DNS resolution and service reachability in-cluster.
Practice what you learned
1. What is the first place to check for the root cause of a Pending pod?
2. What does exit code 137 with reason OOMKilled indicate?
3. How do you view the logs of a container that already crashed and was restarted?
4. What advantage does kubectl debug's ephemeral container offer over kubectl exec?
5. Which is a typical cause of ImagePullBackOff?
Was this page helpful?
You May Also Like
Observability in Kubernetes
Observability in Kubernetes combines metrics, logs, and traces to answer not just whether something is broken, but why, across ephemeral pods and dynamic scheduling.
Kubernetes RBAC
Role-Based Access Control governs who can do what to which Kubernetes resources, using Roles, ClusterRoles, and Bindings to enforce least privilege.
Advanced Kubernetes Interview Questions
A study guide of advanced Kubernetes interview topics spanning scheduling internals, networking, RBAC, and operational judgment expected at senior and staff levels.