Kubernetes Taints, Tolerations and Node Affinity

Welcome to Kubernetes Taints, Tolerations and Node Affinity. We'll break this down from first principles.

Why Default Scheduling Isn't Enough

Kubernetes' default scheduler spreads pods across nodes to balance resource usage. This works for homogeneous clusters but fails when you need to isolate workloads—e.g., GPU nodes for ML training, SSD nodes for databases, or dedicated nodes for PCI-compliant services. Without explicit controls, a burstable pod can evict a critical database pod from a high-performance node. Taints and tolerations solve the repelling side: taint a node so only tolerated pods land there. Node affinity solves the attracting side: express preferences or requirements for node labels. Together, they give you fine-grained control over pod placement, essential for production clusters with mixed workloads.

apiVersion: v1
kind: Pod
metadata:
  name: random-pod
spec:
  containers:
  - name: nginx
    image: nginx:1.25
  nodeSelector:
    disktype: ssd

Taints: Repelling Pods from Nodes

A taint is a key-value pair with an effect applied to a node. The effect can be NoSchedule (don't schedule new pods unless tolerated), PreferNoSchedule (soft version, avoid if possible), or NoExecute (evict existing pods that don't tolerate). Taints are set via kubectl taint nodes <node> key=value:effect. For example, to mark a node for GPU workloads: kubectl taint nodes gpu-node-1 gpu=true:NoSchedule. Pods without a matching toleration will never be scheduled there. This is the foundation of workload isolation. Use NoExecute for critical nodes where you want to evict non-tolerated pods immediately, e.g., for maintenance windows.

kubectl taint nodes node1 dedicated=ml:NoSchedule
kubectl describe node node1 | grep Taints

Tolerations: Allowing Pods on Tainted Nodes

A toleration is a pod specification that matches a taint. It has key, value, effect, and optionally operator (Equal or Exists). If operator is Exists, value is ignored. Tolerations do not schedule pods onto tainted nodes—they only permit scheduling if other constraints (like affinity) also match. For example, a pod with toleration for gpu=true:NoSchedule can land on the GPU node, but it might also land elsewhere. To force it, combine with node affinity or nodeSelector. Tolerations are critical for system pods like kube-proxy or CNI plugins that must run on all nodes, including tainted ones.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-job
spec:
  tolerations:
  - key: "gpu"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
  containers:
  - name: cuda
    image: nvidia/cuda:12.0-base

Node Affinity: Attracting Pods to Nodes

Node affinity is a pod spec field that uses node labels to attract pods. It comes in two types: requiredDuringSchedulingIgnoredDuringExecution (hard requirement) and preferredDuringSchedulingIgnoredDuringExecution (soft preference). The 'ignored during execution' part means if node labels change after scheduling, the pod stays. Use required for critical workloads that must run on specific hardware (e.g., nodes with SSD). Use preferred for performance optimization (e.g., prefer nodes in the same zone). Node affinity supports operators like In, NotIn, Exists, DoesNotExist, Gt, Lt.

apiVersion: v1
kind: Pod
metadata:
  name: db-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: zone
            operator: In
            values:
            - us-east-1a
  containers:
  - name: postgres
    image: postgres:16

Combining Taints, Tolerations, and Node Affinity

For true workload isolation, combine all three: taint nodes to repel unwanted pods, add tolerations to allow your pods, and use node affinity to attract them. This ensures only your pods land on those nodes, and they are forced there. Example: GPU nodes tainted with gpu=true:NoSchedule, ML training pods have toleration for that taint and required node affinity for gpu=true. Without the affinity, pods could still schedule on non-GPU nodes. This pattern is essential for multi-tenant clusters, compliance, and cost allocation.

apiVersion: v1
kind: Pod
metadata:
  name: ml-training
spec:
  tolerations:
  - key: "gpu"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gpu
            operator: In
            values:
            - "true"
  containers:
  - name: trainer
    image: tensorflow/tensorflow:latest-gpu

NoExecute Taint Effect and Pod Eviction

The NoExecute effect evicts all pods that do not tolerate the taint. This is useful for node maintenance or decommissioning. For example, before draining a node, you can add a NoExecute taint to evict non-critical pods gracefully. Pods with toleration can optionally set tolerationSeconds to delay eviction. This is critical for stateful workloads that need time to shut down. Use with caution: evicting pods can cause cascading failures if not handled properly.

kubectl taint nodes node1 maintenance=true:NoExecute
kubectl get pods --field-selector spec.nodeName=node1

Pod Affinity and Anti-Affinity

While node affinity ties pods to nodes, pod affinity/anti-affinity ties pods to other pods. Pod affinity says 'schedule me near pods with label X' (e.g., for low latency). Pod anti-affinity says 'don't schedule me near pods with label Y' (e.g., for high availability). These are expressed as required or preferred, with topologyKey (e.g., kubernetes.io/hostname). Use pod anti-affinity to spread replicas across nodes, and pod affinity to co-locate frontend and backend. Be careful: pod anti-affinity can prevent scheduling if not enough nodes satisfy the constraint.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - web
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: nginx
        image: nginx:1.25

Scheduling Scenarios and Debugging

When pods fail to schedule, check events: kubectl describe pod <pod> shows events like '0/3 nodes are available: 1 node(s) had taint that the pod didn't tolerate, 2 node(s) didn't match node selector'. Common issues: missing tolerations, mismatched taint values, or node affinity expressions that are too restrictive. Use kubectl get nodes -l <label> to verify labels. For taints, kubectl describe node shows all taints. Also check if the node has the correct labels. If using preferred affinity, the scheduler might still schedule elsewhere if no node matches—use required for hard constraints.

kubectl describe pod my-pod | grep -A5 Events
kubectl get nodes --show-labels | grep gpu
kubectl describe node gpu-node | grep Taints

Production Patterns: Dedicated Nodes and System Reservations

In production, use taints and tolerations to reserve nodes for system components (kube-system, monitoring, ingress). Taint all worker nodes with node-role.kubernetes.io/master=true:NoSchedule? No, that's for control plane. Instead, taint nodes with dedicated=system:NoSchedule and add tolerations to system pods. For critical workloads, use a combination of taints and node affinity to guarantee resources. Also consider using PriorityClass to ensure eviction order. Never rely solely on nodeSelector for isolation—it's too weak.

apiVersion: v1
kind: Pod
metadata:
  name: fluentd
  namespace: kube-system
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "system"
    effect: "NoSchedule"
  containers:
  - name: fluentd
    image: fluent/fluentd:v1.16

Advanced: Taint-Based Eviction and TolerationSeconds

TolerationSeconds is a field in toleration that specifies how long a pod can stay on a node after a NoExecute taint is added. This is useful for graceful shutdown of stateful workloads. For example, a database pod might have tolerationSeconds=120 to allow in-flight transactions to complete. After the seconds expire, the pod is evicted. This is a powerful tool for node maintenance without disrupting critical services. However, it requires careful tuning—too short causes abrupt termination, too long delays maintenance.

apiVersion: v1
kind: Pod
metadata:
  name: db-with-grace
spec:
  tolerations:
  - key: "maintenance"
    operator: "Equal"
    value: "true"
    effect: "NoExecute"
    tolerationSeconds: 120
  containers:
  - name: postgres
    image: postgres:16

Common Pitfalls and Anti-Patterns

Pitfall 1: Using only nodeSelector for isolation—it doesn't repel other pods. Pitfall 2: Overusing required anti-affinity, causing unschedulable pods in small clusters. Pitfall 3: Forgetting to add tolerations to DaemonSets, causing them to miss tainted nodes. Pitfall 4: Using the same label for both taint and affinity but with different values—mismatch leads to no schedule. Pitfall 5: Not testing eviction behavior—NoExecute can cause cascading failures. Always test in a staging environment first.

apiVersion: v1
kind: Pod
metadata:
  name: bad-pod
spec:
  nodeSelector:
    gpu: "true"
  containers:
  - name: nginx
    image: nginx

Tooling and Automation

Manage taints and labels via Infrastructure as Code (e.g., Terraform, Pulumi) or Kubernetes operators. For dynamic tainting, use tools like Descheduler or Node Problem Detector. For example, Node Problem Detector can taint a node when it detects disk pressure. Also consider using Kyverno or OPA/Gatekeeper to enforce taint/toleration policies. In CI/CD, validate pod specs with kubeval or conftest. Automate label management with node lifecycle hooks.

resource "kubernetes_node" "example" {
  metadata {
    name = "gpu-node-1"
  }
  spec {
    taint {
      key    = "gpu"
      value  = "true"
      effect = "NoSchedule"
    }
  }
}