FailedScheduling: Fix K8s Pods Stuck Pending Fast

Your deployment looks healthy in Git, the rollout starts, and then... nothing. kubectl get pods shows Pending, and the Events stream fills with FailedScheduling warnings naming predicates like Insufficient cpu, TaintToleration, and NodeAffinity. No container ever starts, so there are no logs to read and no crash to debug. The scheduler tried every node, rejected all of them, and told you exactly why — if you know where to look.

FailedScheduling is uniquely frustrating because it's a decision log, not an error. Each event line explains one reason one node was skipped, and the real cause hides among a dozen near-misses. Teams routinely misread it: they add nodes when requests are bloated, they lower requests when a taint is the blocker, or they chase zone labels while an unbound PVC holds the pod hostage.

This guide teaches you to read the scheduler's verdict the way it was meant to be read. You'll start with per-node event reasons, then work through the six real causes in frequency order: resource insufficiency, taints and tolerations, selectors and affinity, unbound PVCs with zone mismatch, and topology spread math. By the end you'll clear most Pending pods with three commands and stop the rest with constraints you've actually done the math on.

Every FailedScheduling investigation starts in exactly one place: the Events section of kubectl describe pod. The scheduler logs a Warning per scheduling cycle naming each predicate that failed, and — critically — it logs per node. You'll see lines like 0/14 nodes available: 9 Insufficient cpu, 5 TaintToleration, and that ratio is the whole diagnosis. The majority reason is your cause; the minority lines are nodes that failed for their own unrelated reasons. Engineers who grep logs cluster-wide miss this because they're reading aggregated noise instead of one pod's verdict.

Learn to separate the scheduler's two decision phases. Predicates are hard filters — a node failing any predicate is out, no scoring involved. Priority scoring only ranks the survivors, so it never appears in FailedScheduling at all. That means every event line is a veto, and vetoes compose: a node can fail taint tolerance and capacity simultaneously while the event counts it once. When multiple predicates appear, fix the one covering the most nodes first, then re-describe — the next cycle's events will reveal whatever was hiding underneath.

Make describe-pod-events your reflex before any YAML edit or scale-up. The command costs seconds and has prevented countless wasted node additions: if 9 of 14 nodes say TaintToleration, no amount of new capacity schedules onto the tainted majority. Read the ratio, fix the majority veto, and let the next scheduling cycle tell you what's left.

# The single highest-value command for Pending pods
kubectl get pods -A | grep -v Running
kubectl describe pod <pod-name> -n <namespace> | sed -n '/Events:/,$p'

# Watch the verdict change as you fix things
kubectl get events --sort-by=.lastTimestamp -n <namespace> | grep FailedScheduling

# Count rejections per reason across recent cycles
kubectl describe pod <pod-name> -n <namespace> | grep -oE '[0-9]+ [A-Za-z]+' | sort | uniq -c

Insufficient cpu and Insufficient memory are the most common FailedScheduling lines and the most misread. They don't mean your cluster is busy — they mean the sum of container requests on each node leaves no room for your pod's requests. Requests are reservations deducted from allocatable capacity at schedule time, whether the containers use them or not. A pod requesting 4 CPUs while burning 300m consumes 4 CPUs of scheduling ledger and 300m of silicon, and the scheduler only cares about the ledger.

The gap between reserved and used is where phantom capacity crises live. Teams copy request values from load-test manifests, double them for safety, and never revisit — until routine pods can't schedule on nodes running at 18 percent real utilization. Memory is harsher than CPU: CPU throttles past limits, but memory requests that exceed allocatable block scheduling outright, and memory limits that get breached kill with OOM. Both show up as scheduling vetoes long before any runtime symptom.

Fix it with measurement, not Spares. Compare Allocated against Allocatable in kubectl describe node to see the ledger, then find the hogs with a sorted kubectl top pods. Set requests near measured p95 usage — 800m for a pod burning 700m — and reserve headroom at the node-pool level instead of per pod. Right-sizing 41 launch pods from 4 CPUs to 800m freed the equivalent of 11 nodes without spending a dollar. Add capacity only when the ledger is honest and still full.

# The scheduling ledger: what nodes really have left
kubectl describe node <node> | grep -A 6 'Allocated resources'

# Real usage versus reservations, cluster-wide
kubectl top nodes
kubectl top pods -A --sort-by=cpu | head -15
kubectl top pods -A --sort-by=memory | head -15

# Find the most over-provisioned requests
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}{" cpu_req="}{.spec.containers[0].resources.requests.cpu}{"\n"}{end}' | head -20

Taints are the scheduler's keep-out signs: any node carrying a taint rejects every pod that doesn't declare a matching toleration. They appear after cordons, GPU and spot-node provisioning, and upgrades — and they linger. A NoSchedule taint applied during Thursday's upgrade survives on any node whose cleanup job failed, quietly shrinking your schedulable fleet while dashboards show all nodes Ready. Ready means the kubelet is healthy; it says nothing about whether the scheduler is allowed to use the node.

Matching is exact and unforgiving. A toleration must agree on key, value, and effect — effect NoSchedule doesn't match NoExecute, and a missing value doesn't match a valued taint. Teams stare at a toleration that looks right while one field differs, then conclude taints aren't the problem. The describe output shows both sides if you look: node Taints at the top of kubectl describe node, pod Tolerations in the pod spec. Diff them character by character before theorizing further.

Treat taints as incident debris to be cleaned, not as permanent architecture. After every upgrade, cordon, and GPU job, audit taints across the fleet and remove the ones whose reason expired. Bake the audit into runbooks: the nodes that reboot mid-cleanup are exactly the ones that keep their taints. And when a taint is legitimate — spot nodes, dedicated hardware — put the toleration in the deployment template once, not as tribal knowledge engineers re-derive each outage.

tolerations:
- key: "upgrade.in.progress"
  operator: "Exists"
  effect: "NoSchedule"
- key: "nvidia.com/gpu"
  operator: "Equal"
  value: "present"
  effect: "NoSchedule"

Node selectors and affinity rules tell the scheduler which nodes are acceptable, and a required rule that matches nothing vetoes the entire fleet. The failure modes are mundane: a label typo like zone-a versus zone_a, a label that existed on the old node pool but not the new one, or an In operator listing values nobody applied. The pod spec looks deliberate, the intent was reasonable six months ago, and today zero nodes qualify — so the pod pends with a perfectly logical explanation nobody re-examined.

Required versus preferred is the whole game. requiredDuringSchedulingIgnoredDuringExecution is a veto: miss it and the pod never schedules, full stop. preferredDuringSchedulingIgnoredDuringExecution is a wish: the scheduler tries, fails soft, and places the pod anyway. Teams reach for required when they want tidiness — keep the API near the cache — and buy fragility: one relabeled node pool and the tier can't schedule anywhere. Affinity also compounds across terms: nodeAffinity plus podAntiAffinity plus spread constraints must all pass on the same node, and each added term multiplies the ways to get zero.

Defend with verification before constraint. List real labels with kubectl get nodes --show-labels and write selectors against what exists, not what you remember. Default to preferred for placement wishes and reserve required for genuine hard needs like GPU models or compliance zones. When a pod pends on affinity, comment out terms one at a time and re-describe — the cycle that flips to Scheduled names the guilty term faster than any amount of YAML staring.

# Which side of the binding is stuck?
kubectl get pvc -A | grep -v Bound
kubectl get pv -o yaml | grep -B 2 -A 6 nodeAffinity

# StorageClass binding mode decides who moves first
kubectl get storageclass -o yaml | grep -E 'name:|volumeBindingMode|allowedTopologies' 

# Pod's zone demands for comparison
kubectl get pod <pod> -o yaml | grep -A 8 'nodeAffinity\|topologyKey'