Kubernetes Node Management — Cordon, Drain, Upgrades

Welcome to Kubernetes Node Management — Cordon, Drain, Upgrades. We'll break this down from first principles.

Understanding Node States: Ready, NotReady, and SchedulingDisabled

A Kubernetes node can be in several states that affect workload scheduling. The Ready status indicates the node is healthy and can accept pods. NotReady means the node is unreachable or unhealthy — pods may be evicted after a timeout. SchedulingDisabled is a manual state set by cordon that prevents new pods from being scheduled while existing pods continue running. Understanding these states is critical for planning maintenance. When you cordon a node, its spec.unschedulable field is set to true, which the scheduler respects. The node remains Ready but won't accept new pods. This is the first step in a safe drain sequence.

kubectl get nodes -o wide
# Output:
# NAME           STATUS                     ROLES    AGE   VERSION
# node-1         Ready,SchedulingDisabled   worker   10d   v1.28.3
# node-2         Ready                      worker   10d   v1.28.3
# node-3         NotReady                   worker   10d   v1.28.3

Cordon: The Safety Gate Before Maintenance

The cordon command marks a node as unschedulable. It's the first step in any node maintenance procedure because it prevents new pods from landing on the node while existing workloads continue. This is crucial for zero-downtime operations. Without cordoning, new pods could be scheduled during the drain, causing race conditions. Cordon is idempotent — running it multiple times has no side effect. Use it before any planned maintenance, even if you're not draining immediately. It gives you a safety window to verify the node's health and plan the drain. The command updates the node's spec.unschedulable field to true.

kubectl cordon node-1
# Output: node/node-1 cordoned

# Verify:
kubectl get node node-1 -o jsonpath='{.spec.unschedulable}'
# Output: true

Drain: Evicting Pods Gracefully

The drain command evicts all pods from a node while respecting PodDisruptionBudgets (PDBs) and terminationGracePeriodSeconds. It first cordons the node (if not already), then evicts pods one by one. DaemonSet pods are not evicted by default — you must use --ignore-daemonsets or --delete-emptydir-data flags. Drain waits for PDBs to allow at least one replica to remain available. If a PDB blocks eviction, drain will hang until timeout. This is intentional to protect application availability. Use --force to bypass PDBs (dangerous in production). Drain also respects pod priority — higher priority pods are evicted last.

kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data --timeout=300s
# Output:
# node/node-1 cordoned
# evicting pod "nginx-abc123"
# evicting pod "api-server-def456"
# ...
# node/node-1 drained

Uncordon: Returning the Node to Service

After maintenance, uncordon marks the node as schedulable again. It sets spec.unschedulable to false. Pods are not automatically rescheduled onto the node — they will be scheduled only when new pods are created or when existing pods are rescheduled due to other reasons. This is a common misconception. If you want to rebalance workloads, you may need to manually delete pods on other nodes or use descheduler. Uncordon is safe to run at any time, even if the node is not fully ready. However, ensure the node's kubelet is healthy and all system components are running before uncordoning.

kubectl uncordon node-1
# Output: node/node-1 uncordoned

# Verify:
kubectl get node node-1 -o jsonpath='{.spec.unschedulable}'
# Output: false

Drain Flags and Options You Must Know

The kubectl drain command has several flags that control behavior. --ignore-daemonsets is almost always needed because DaemonSet pods are not evicted by default. --delete-emptydir-data allows eviction of pods using emptyDir volumes — without it, drain will refuse to evict such pods. --force bypasses PDBs and termination grace periods — use only as a last resort. --grace-period sets the duration for pod termination. --timeout specifies how long to wait before giving up. --pod-selector allows draining only pods matching a label. Understanding these flags prevents unexpected failures. For example, forgetting --ignore-daemonsets will cause drain to hang indefinitely.

kubectl drain node-1 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=300s \
  --force=false

PodDisruptionBudget: The Gatekeeper of Availability

PodDisruptionBudget (PDB) is a Kubernetes resource that limits the number of pods that can be voluntarily disrupted at a time. It ensures that a minimum number of pods remain available during voluntary disruptions like node drains. PDBs are defined per application and specify minAvailable or maxUnavailable. When draining a node, the eviction API checks PDBs. If evicting a pod would violate the PDB, the eviction is rejected. This can cause drain to hang. To avoid this, ensure your PDBs are configured with enough buffer. For example, if you have 3 replicas, set maxUnavailable: 1 to allow one pod to be disrupted at a time.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: nginx

Node Upgrades: A Step-by-Step Workflow

Upgrading a Kubernetes node (e.g., kubelet, container runtime, or OS) requires a careful workflow to avoid downtime. The standard process: 1) Cordon the node to prevent new pods. 2) Drain the node to evict existing pods. 3) Perform the upgrade (e.g., package update, reboot). 4) Verify node health (kubelet, node status). 5) Uncordon the node. This sequence ensures zero pod disruption. For control plane nodes, the process is similar but requires special attention to etcd and API server quorum. Always upgrade one node at a time, especially in production. Automate this with tools like kubeadm upgrade node or cluster-api.

# Step 1: Cordon
kubectl cordon node-1

# Step 2: Drain
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

# Step 3: Upgrade (example for Ubuntu)
ssh node-1 'sudo apt update && sudo apt upgrade -y kubelet kubeadm kubectl && sudo systemctl restart kubelet'

# Step 4: Verify
kubectl get node node-1

# Step 5: Uncordon
kubectl uncordon node-1

Automating Node Management with Scripts and Tools

Manual node management doesn't scale. In production, automate cordon/drain/uncordon with scripts or tools like Ansible, Terraform, or cluster-api. A common pattern is to write a bash script that iterates over nodes, cordons, drains, performs maintenance, verifies, and uncordons. Use kubectl with --context for multi-cluster support. For cloud-managed clusters (EKS, AKS, GKE), use their node management features (e.g., AWS Node Termination Handler). For on-prem, consider using a tool like kured (Kubernetes Reboot Daemon) that automates node reboots based on reboot-required files.

#!/bin/bash
set -euo pipefail

NODE=$1

echo "Cordoning $NODE"
kubectl cordon "$NODE"

echo "Draining $NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=300s

echo "Performing maintenance on $NODE"
ssh "$NODE" 'sudo apt upgrade -y kubelet && sudo systemctl restart kubelet'

echo "Waiting for node to be Ready"
until kubectl wait --for=condition=Ready node/"$NODE" --timeout=60s; do
  sleep 5
done

echo "Uncordoning $NODE"
kubectl uncordon "$NODE"
echo "Done"

Handling Edge Cases: When Drain Fails

Drain can fail for several reasons: PDB violations, pods with local storage (emptyDir without --delete-emptydir-data), unmanaged pods (not backed by a controller), or pods with strict inter-pod anti-affinity. When drain fails, it will output the reason. Common fixes: adjust PDBs, add --delete-emptydir-data, or delete unmanaged pods manually. For pods with anti-affinity, you may need to temporarily scale down the deployment. In extreme cases, use --force but be aware of the risks. Always investigate the root cause rather than blindly forcing. A failed drain is a signal that your cluster configuration needs attention.

# Drain fails with:
kubectl drain node-1 --ignore-daemonsets
# error: unable to drain node "node-1", aborting command...
# There are pending pods in node "node-1":
#   pod "my-pod" (unmanaged)

# Solution: Delete the unmanaged pod
kubectl delete pod my-pod --grace-period=0 --force
kubectl drain node-1 --ignore-daemonsets

Best Practices for Production Node Management

1. Always cordon before drain, even if drain does it automatically. 2) Use PDBs with maxUnavailable: 1 for all stateless workloads. 3) Set terminationGracePeriodSeconds appropriately (e.g., 30s for web apps, 120s for batch jobs). 4) Use --ignore-daemonsets and --delete-emptydir-data in all drain commands. 5) Test drain on a non-production cluster first. 6) Monitor cluster during maintenance with alerts on node status and pod evictions. 7) Have a rollback plan: if a node fails to come back after upgrade, have a process to replace it. 8) Use cluster autoscaler to handle capacity during node drains.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: nginx
        image: nginx:1.25
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: web

Troubleshooting Common Node Management Issues

Common issues: Node stuck in NotReady after reboot — check kubelet logs (journalctl -u kubelet). Drain hangs — check PDBs and pod eviction events (kubectl get events --field-selector involvedObject.kind=Pod). Uncordon doesn't schedule pods — check node conditions and taints. Pods evicted but not rescheduled — check scheduler logs or node resources. Always use kubectl describe node to get a full picture. For persistent issues, consider replacing the node rather than debugging. In cloud environments, terminating and recreating the node is often faster.

# Check node conditions
kubectl describe node node-1 | grep -A5 Conditions

# Check kubelet logs
ssh node-1 'journalctl -u kubelet --since "5 minutes ago"'

# Check eviction events
kubectl get events --field-selector involvedObject.kind=Pod --sort-by='.lastTimestamp'

# Check taints
kubectl get node node-1 -o jsonpath='{.spec.taints}'

Advanced: Using Descheduler for Workload Rebalancing

After uncordoning a node, pods don't automatically move to it. The descheduler is a Kubernetes add-on that evicts pods based on policies to rebalance workloads. For example, after a node returns from maintenance, the descheduler can evict pods from other nodes to fill the empty node. This is useful for optimizing resource utilization. However, descheduler must be used carefully — it can cause unnecessary pod churn. Configure it with policies like LowNodeUtilization and HighNodeUtilization. Always run descheduler in dry-run mode first. In production, we run descheduler every 15 minutes with a MaxNoOfPodsToEvictPerNode limit.

apiVersion: "descheduler/v1alpha2"
kind: "DeschedulerPolicy"
profiles:
  - name: LowNodeUtilization
    pluginConfigs:
    - name: "LowNodeUtilization"
      args:
        thresholds:
          cpu: 20
          memory: 20
        targetThresholds:
          cpu: 50
          memory: 50
    plugins:
      deschedule:
        enabled:
          - "LowNodeUtilization"