Attracting Pods to Nodes: Node Affinity
Node affinity is a Pod-side rule that constrains which nodes a Pod can be scheduled onto, based on labels attached to nodes. It comes in two flavors: requiredDuringSchedulingIgnoredDuringExecution, a hard constraint the scheduler must satisfy (the Pod stays Pending if unsatisfiable), and preferredDuringSchedulingIgnoredDuringExecution, a soft constraint expressed as a weighted list that the scheduler tries to honor but will ignore if no matching node has capacity. The 'IgnoredDuringExecution' suffix on both means Kubernetes never evicts a running Pod if node labels change after scheduling; affinity is only evaluated at placement time, not continuously.
Cricket analogy: It is like a team management rule that a certain all-rounder must be selected only for pitches labeled 'spin-friendly' (a hard requirement), versus a softer preference to bat a certain player at number four whenever the ground allows it, but not something that gets them substituted mid-innings if conditions change.
Repelling Pods from Nodes: Taints and Tolerations
Taints are the inverse mechanism: they are applied to nodes, not Pods, and by default repel every Pod from scheduling there unless the Pod carries a matching toleration. A taint has a key, value, and effect: NoSchedule prevents new Pods from being scheduled but does not touch Pods already running; PreferNoSchedule is a soft version the scheduler tries to avoid but will violate under pressure; and NoExecute both prevents new scheduling and actively evicts already-running Pods that lack a matching toleration, optionally after a tolerationSeconds grace period. This asymmetry (Pods opt in to nodes via affinity, nodes opt out Pods via taints) is deliberate: it lets a cluster operator dedicate nodes (like GPU or spot-instance nodes) without every unrelated team's Deployment needing to know that node exists.
Cricket analogy: It is like a ground curator marking a pitch as 'off-limits for practice' so no team can use it unless they hold special net-booking clearance, versus a stricter version where even a team currently practicing there gets asked to leave once the ground is reclassified for match preparation, mirroring NoExecute.
# Node: taint dedicated GPU nodes so only GPU workloads land there
# kubectl taint nodes gpu-node-1 nvidia.com/gpu=true:NoSchedule
apiVersion: v1
kind: Pod
metadata:
name: gpu-inference
spec:
tolerations:
- key: "nvidia.com/gpu"
operator: "Equal"
value: "true"
effect: "NoSchedule"
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node.kubernetes.io/instance-type
operator: In
values: ["g5.xlarge", "g5.2xlarge"]
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values: ["us-east-1a"]
containers:
- name: infer
image: my-registry/infer:latest
resources:
requests:
nvidia.com/gpu: "1"Combining Both: Dedicating and Isolating Nodes
Taints alone only prevent unwanted Pods from landing on a node; they do not guarantee that the Pods you do want will land there instead, since a GPU-tolerating Pod could still be scheduled onto an ordinary untainted node. The standard pattern for truly dedicating a node pool is to combine a taint on the nodes with both a matching toleration and a node affinity or nodeSelector on the intended Pods, so the taint keeps everyone else out while the affinity/selector pulls the right workload in. Kubernetes also has built-in NoExecute taints the control plane applies automatically, such as node.kubernetes.io/not-ready and node.kubernetes.io/unreachable, which is why Pods have a default tolerationSeconds of 300 for these, giving a node a five-minute grace period to recover before its Pods are evicted and rescheduled elsewhere.
Cricket analogy: It is like a ground reserving a specific net purely for pace bowlers via a sign (the taint) but also actively assigning pace bowlers to that net on the roster (the affinity), because the sign alone would not stop a spinner from wandering in and the roster alone would not stop others from wandering in either.
kubectl describe node <name> shows both Taints and Labels; kubectl get pod <name> -o yaml shows a Pod's tolerations and affinity rules. When debugging a Pod stuck Pending, checking these two together (via kubectl describe pod, which surfaces FailedScheduling events explaining exactly which predicate failed) is the fastest path to a root cause.
Node affinity with requiredDuringSchedulingIgnoredDuringExecution is evaluated only at scheduling time. If a node's labels change after a Pod is already running there (for example, a label is removed by an operator), the Pod is NOT evicted — it keeps running on a node it would no longer be eligible for. Do not rely on affinity for continuous enforcement; use it only for placement decisions.
- Node affinity is a Pod-side pull toward nodes with matching labels; taints are a node-side push away from Pods without matching tolerations.
- requiredDuringScheduling is a hard constraint; preferredDuringScheduling is a soft, best-effort constraint.
- Taint effects: NoSchedule (block new Pods), PreferNoSchedule (soft block), NoExecute (block new Pods and evict running ones).
- IgnoredDuringExecution means affinity rules are checked only at scheduling time, never enforced continuously on running Pods.
- To truly dedicate a node pool, combine a taint (keep others out) with a matching toleration plus affinity/selector (pull the right Pods in) on the same nodes/Pods.
- Kubernetes auto-applies NoExecute taints like node.kubernetes.io/not-ready; default tolerationSeconds of 300 gives nodes a grace period before eviction.
- kubectl describe pod surfaces FailedScheduling events that name the exact predicate (taint or affinity) blocking placement.
Practice what you learned
1. What is the fundamental directional difference between node affinity and taints?
2. Which taint effect will evict Pods that are already running on a node and lack a matching toleration?
3. If a node's label is changed after a Pod with requiredDuringSchedulingIgnoredDuringExecution affinity is already running there, what happens?
4. What is the correct pattern for truly dedicating a node pool to a specific workload?
5. What is the default tolerationSeconds Kubernetes applies to Pods for the node.kubernetes.io/not-ready and node.kubernetes.io/unreachable taints?
Was this page helpful?
You May Also Like
Cluster Autoscaling
Learn how the Cluster Autoscaler adds and removes worker nodes so Pod-level autoscalers always have capacity to grow into.
Resource Requests and Limits
Understand how Kubernetes uses requests for scheduling and limits for runtime enforcement, and how both determine a Pod's QoS class.
Horizontal Pod Autoscaling
Learn how Kubernetes automatically scales the number of Pod replicas in and out based on CPU, memory, and custom metrics.