Kubernetes PodDisruptionBudget: Stop Losing Pods During Rolling Updates

You've seen it: a rolling update or node drain takes down all your pods at once, and your service goes dark. Kubernetes gives you the tools to prevent this, but most teams ignore them until it's too late. PodDisruptionBudgets are the difference between a graceful maintenance and a PagerDuty alert at 3 AM.

The problem is simple: voluntary disruptions — node drains, cluster upgrades, autoscaling down — can evict pods. Without a PDB, the eviction controller doesn't care about your app's availability. It'll drain a node hosting all your replicas, and boom, downtime.

By the end of this, you'll know exactly how to configure PDBs for zero-downtime maintenance, how to debug them when they block an operation, and the one gotcha that makes people rip their hair out.

What Exactly Is a Voluntary Disruption?

Kubernetes splits disruptions into two types: voluntary and involuntary. Involuntary disruptions are hardware failures, kernel panics, OOM kills — things you can't control. Voluntary disruptions are actions you or the cluster take: node drains, cluster autoscaler scale-down, rolling updates, pod evictions via API. A PDB only protects against voluntary disruptions. If a node dies, your pods are gone — PDB won't save you. But for everything else, it's your safety net.

Without a PDB, the eviction controller treats each pod independently. It doesn't know your app needs at least 2 replicas. So when you drain a node, it evicts all pods on that node, even if that means your app goes to zero. That's the bug.

PDBs work by intercepting eviction requests. When you drain a node or delete a pod, the eviction API checks the PDB. If evicting the pod would drop the available count below minAvailable (or exceed maxUnavailable), the eviction is rejected. The drain pauses, and the controller waits until a new pod is ready elsewhere.

// io.thecodeforge — DevOps tutorial

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-api-pdb
spec:
  minAvailable: 2  # At least 2 pods must always be running
  selector:
    matchLabels:
      app: payments-api

minAvailable vs maxUnavailable: When to Use Which

You have two knobs: minAvailable and maxUnavailable. They're mutually exclusive — pick one. minAvailable specifies the absolute number or percentage of pods that must remain available. maxUnavailable specifies how many can be unavailable. For a 3-replica deployment, minAvailable: 2 is equivalent to maxUnavailable: 1.

Use minAvailable when you have a hard floor on capacity. For example, a database read replica pool must always have at least 3 replicas to handle query load. Use maxUnavailable when you know your burst tolerance. For a web app that can handle a 50% traffic drop briefly, maxUnavailable: 50% is fine.

The percentage is calculated from the current number of replicas, not the desired. If you scale down from 10 to 5, maxUnavailable: 50% allows 5 pods down at the start, but as pods are removed, the percentage shrinks. This can cause unexpected blocking. I've seen a scale-down get stuck because maxUnavailable: 50% of 5 is 2.5 (rounded up to 3), but after evicting 3, only 2 remain, and the next eviction would violate the budget. The fix: use absolute numbers for critical apps.

// io.thecodeforge — DevOps tutorial

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-frontend-pdb
spec:
  maxUnavailable: 1  # Never more than 1 pod down at a time
  selector:
    matchLabels:
      app: web-frontend

How PDBs Interact with Rolling Updates and Node Drains

A common misconception: the Deployment's maxSurge and maxUnavailable in the update strategy control all disruptions. They don't. Those only apply during a rolling update triggered by a Deployment change. Node drains, cluster autoscaler scale-down, and manual evictions bypass the Deployment's strategy entirely. That's where PDBs come in.

When you kubectl drain a node, the eviction API checks the PDB for each pod on that node. If evicting a pod would violate the PDB, the drain pauses. The node gets marked as SchedulingDisabled, and the drain command waits. It won't time out — it'll sit there until the PDB allows the eviction. This is often what people see when a drain hangs.

Rolling updates also respect PDBs. The Deployment controller creates new pods, waits for them to be ready, then evicts old pods. But the eviction of old pods goes through the eviction API, which checks the PDB. If the PDB says maxUnavailable: 1, the rolling update will only evict one old pod at a time, even if the Deployment strategy allows more. This is usually fine, but it can slow down updates. If you need speed, increase maxUnavailable temporarily.

// io.thecodeforge — DevOps tutorial

# Drain a node with a PDB protecting the pods
kubectl drain node-1 --ignore-daemonsets
# Output will show eviction attempts and possibly wait
# If PDB blocks, you'll see:
#   evicting pod default/payments-api-xxx
#   error when evicting pods: Cannot evict pod as it would violate the pod's disruption budget.
# The drain will retry indefinitely until a new pod is ready.

# To force drain (skip PDB check) — use with caution:
kubectl drain node-1 --ignore-daemonsets --disable-eviction

PDB and Cluster Autoscaler: The Silent Scale-Down Blocker

Cluster Autoscaler (CA) scales down nodes when they're underutilized. It evicts pods from those nodes before removing them. If a PDB blocks the eviction, CA can't scale down that node. This leads to nodes sitting idle, wasting money.

I've seen this in production: a team had a PDB with minAvailable: 3 on a 3-replica deployment. CA tried to scale down a node that hosted one of those replicas. The eviction was blocked because removing that pod would leave only 2 available, violating the PDB. CA gave up, and the node stayed. The fix: either set minAvailable: 2 or use maxUnavailable: 1 to allow one pod to be evicted.

Another gotcha: CA uses a different eviction mechanism than kubectl drain. It respects PDBs the same way, but it has a timeout. If the eviction is blocked for too long (default 10 minutes), CA moves on. This can leave pods stranded on a node that should have been scaled down. Monitor CA logs for skip node ... - pod ... has a disruption budget that prevents eviction.

// io.thecodeforge — DevOps tutorial

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: worker-pdb
spec:
  maxUnavailable: 1  # Allow CA to evict one pod at a time
  selector:
    matchLabels:
      app: worker

Unhealthy Pod Eviction: The PDB Bypass You Need to Know

There's a critical exception: PDBs do not protect unhealthy pods. If a pod has a status condition like Ready=False or has been in Terminating for more than a few minutes, the eviction controller may evict it even if it would violate the PDB. This is by design — you don't want a broken pod blocking a drain.

But here's the trap: the definition of 'unhealthy' is strict. The pod must have a status.conditions entry with type: Ready and status: False. If the pod is simply not ready (e.g., liveness probe failing but condition not updated), it's still considered healthy by the eviction controller. I've seen a pod with a failing liveness probe block a drain for 5 minutes because the condition wasn't set to False.

To force eviction of a stuck pod, you can delete it directly with kubectl delete pod --force --grace-period=0. This bypasses the eviction API entirely. Use this when a pod is truly dead and blocking operations.

// io.thecodeforge — DevOps tutorial

# Force delete a pod that's blocking a drain
kubectl delete pod payments-api-xxx --force --grace-period=0
# This bypasses the eviction API and PDB check

Debugging PDBs: Why Is My Drain Stuck?

When a drain hangs, the first thing to check is the PDB status. Run kubectl get pdb -n <namespace>. Look at the DISRUPTIONS ALLOWED column. If it's 0, the PDB is blocking evictions. Then check EXPECTED PODS and CURRENT HEALTHY — if they're equal, no pod can be evicted without violating the budget.

Next, check which pods are on the draining node: kubectl get pods --field-selector spec.nodeName=<node>. See if any of those pods match the PDB selector. If they do, you need to either increase maxUnavailable, scale up the deployment, or force the drain.

Another common issue: the PDB selector doesn't match any pods. This happens when labels change. The PDB will show EXPECTED PODS: 0, meaning it's a no-op. You think you're protected, but you're not. Always verify with kubectl describe pdb <name>.

// io.thecodeforge — DevOps tutorial

# Check PDB status
kubectl get pdb -n default
# Example output:
# NAME              MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
# payments-api-pdb   2               N/A               1                     10m

# Describe PDB for details
kubectl describe pdb payments-api-pdb
# Shows selector, status, and which pods match

# List pods on a specific node
kubectl get pods --field-selector spec.nodeName=node-1 -o wide

When Not to Use a PDB

PDBs aren't free. They add latency to drains and rolling updates. For stateless batch jobs that can be interrupted, skip the PDB. For development clusters, skip it — you want fast iteration, not safety nets. For single-replica deployments, a PDB with minAvailable: 1 is pointless because evicting that one pod would always violate it, blocking all drains. Either run multiple replicas or accept downtime during maintenance.

Also, don't use PDBs for DaemonSets. DaemonSet pods are tied to nodes — draining a node will evict the DaemonSet pod regardless. PDBs don't apply to DaemonSets in practice because the eviction controller treats them specially (they're allowed to be evicted even with a PDB). Check the docs, but I've seen this confuse people.