Kubernetes Unbound PVC: Fix Pending Pods

Your Deployment is applied, the pods exist, and every one sits in Pending with "0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims." No container started, no logs exist, and kubectl logs returns nothing because there's nothing to log. Storage failures are uniquely frustrating: the workload is correct, the images are fine, and the cluster refuses to schedule over paperwork — a claim with no matching volume.

The error compresses several distinct causes into one sentence. The StorageClass may not exist (a typo, or a class from another cloud's tutorial). The claim may ask for more than any volume offers, or an access mode the driver can't provide. Binding may be delayed by design (WaitForFirstConsumer) while you expected instant. Or dynamic provisioning is broken — the provisioner pod is missing, misconfigured, or erroring — so no volume ever gets created.

This guide works the chain in order: read the PVC events, verify class and static volumes, reconcile size and access modes, understand the binding mode, and debug the provisioner. You'll get the kubectl commands that reveal each link and manifests that bind correctly.

Start at the Claim: Events Name the Cause

kubectl describe pod tells you the pod is waiting; kubectl describe pvc tells you why. The claim's Events section is the single richest diagnostic in storage debugging: the controller writes exactly what blocked binding — a StorageClass it can't find, a provisioner call that failed with the cloud error attached, or a static match that found no candidates. Teams that start at the pod burn hours on node and image theories; teams that start at the claim read the answer in thirty seconds. Make describe pvc your reflex for every Pending pod with a volume.

Pair it with the wide view: kubectl get pvc shows phase (Pending vs Bound), age (how long it's waited), and volume (empty until bound). A claim Pending for 3 hours with FailedProvisioning events repeating is a provisioner problem; Pending with no events at all suggests the class doesn't exist (no controller watches it) or the binding mode is waiting on scheduling. kubectl get events --sort-by=.lastTimestamp -n <ns> | grep -i -E 'pvc|provision|volume' gives the namespace timeline when multiple claims fail together.

Practice reading the three canonical messages until they're instant pattern matches. storageclass.storage.k8s.io X not found means fix the name. no persistent volumes available means static binding found nothing (check the PV table). ProvisioningFailed with a cloud error means the driver called the API and lost — read the attached message, not the prefix. Each message maps to exactly one section below.

StorageClass Names: Spelling Is the API

The storageClassName field is a strict string match against a cluster-scoped object — one wrong character and no provisioner engages, with no fuzzy matching and no helpful suggestion. Cross-environment copies are the top source: fast-ssd-prod in a cluster that defines fast-ssd, gp2 from an AWS tutorial applied to GKE, or a Helm values file overriding the class per environment with a stale entry. kubectl get sc lists the actual vocabulary; the claim must use exactly one of those words or nothing happens.

Mind the empty-string edge. A claim with no storageClassName requests the cluster's default class (the one annotated storageclass.kubernetes.io/is-default-class). If no default exists, an empty class means static-only binding — dynamic provisioning silently never starts. Conversely, an explicit class on the claim and a different (or empty) class on a static PV never bind: both sides must name the same class or both omit it. Check both objects, not just the claim.

Fixes are small and permanent. Correct the name in the manifest or values file, set a default class per cluster so bare claims provision, and standardize class names across environments so overlays can't diverge. The pipeline gate from the incident — every rendered storageClassName must exist in the target cluster — turns this entire section into a CI failure instead of a 3 AM page. Spelling is the API; lint it like one.

# The cluster's actual vocabulary
kubectl get sc -o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner,MODE:.volumeBindingMode,DEFAULT:.metadata.annotations

# What the claim asks for (class, size, modes in one line)
kubectl get pvc data-postgres-0 -n staging -o jsonpath='class={.spec.storageClassName} size={.spec.resources.requests.storage} modes={.spec.accessModes}{"\n"}'

# Which class is the default? (empty claim class binds here)
kubectl get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.metadata.name}{"\n"}{end}'

# Pipeline gate: every rendered class must exist in the target cluster
# for c in $(grep -rhoP 'storageClassName:\s*\K\S+' rendered/); do
#   kubectl get sc "$c" >/dev/null || { echo "missing class: $c"; exit 1; }
# done

Static Binding: Capacity, Modes, and the PV Table

Without dynamic provisioning, binding is a three-way match between claim and a pre-created PersistentVolume: the volume's capacity must be greater than or equal to the request (a 5Gi volume never satisfies a 10Gi claim, and Kubernetes won't partially fill), the claim's accessModes must be a subset of the volume's (claiming ReadWriteMany against RWO-only volumes fails), and storageClassName must agree on both sides. One mismatch and the claim waits forever — silently, since static matching emits few events. Read the PV table, not your assumptions.

Reclaim policy and phase add two more traps. A volume in Released phase (previously bound, claim deleted without wiping the volume) never rebinds automatically — it needs manual reclaim (retain-and-clean) or deletion and recreation. And accessModes semantics are driver-real: ReadWriteOnce means one node, so a claim needing multi-node writers genuinely needs RWX-capable storage (NFS, EFS, CephFS), not a bigger EBS volume. Asking EBS for RWX is asking the impossible, however the manifest phrases it.

The one-table diagnosis: kubectl get pv with custom columns for class, capacity, modes, and phase shows every candidate and its disqualifier at a glance. Fix the actual column — resize the request, align the modes with what the driver supports, match the class names, or recycle Released volumes. Static binding rewards table reading; every other approach is guessing.

# A static PV and the claim that binds to it: class, size, modes agree.
# kubectl get pv -o custom-columns shows disqualifiers at a glance.
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-uploads-10gi
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: manual
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /mnt/data-uploads
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: uploads
  namespace: staging
spec:
  storageClassName: manual   # must equal the PV's class (or both empty)
  accessModes:
    - ReadWriteOnce          # subset of the PV's modes
  resources:
    requests:
      storage: 5Gi           # <= the PV's 10Gi capacity

WaitForFirstConsumer vs Immediate: Timing, Not Failure

VolumeBindingMode splits provisioning into two philosophies. Immediate binds at claim creation — the volume exists before any pod, which is simple but zone-blind: the provisioner may carve storage in zone A while the pod later schedules in zone B, stranding both. WaitForFirstConsumer waits for a pod to need the claim, then provisions near the chosen node — correct topology, but the claim reads Unbound and the pod waits during the delay. Newcomers misread that wait as breakage and start deleting things that were working.

Know which mode your class uses before debugging: kubectl get sc <class> -o jsonpath for volumeBindingMode. Immediate plus unbound means a real failure (provisioner error or no candidates) — dig in. WaitForFirstConsumer plus unbound means scheduling is mid-flight — check the pod's scheduler events for the actual constraint (node affinity, taints, zone limits) rather than the storage. The storage resolves itself once scheduling picks a home.

Choose deliberately per workload. Stateful sets with node affinity and multi-zone clusters want WaitForFirstConsumer (it's the default in most provisioners' recommended classes). Single-zone dev clusters and pre-provisioned static volumes are fine with Immediate. Document the mode per class in your cluster README so the next on-call doesn't debug a normal delay — and set the Pending alert threshold past the normal bind window so it pages for stuck, not for slow.

# Which timing philosophy does your class use?
kubectl get sc fast-ssd -o jsonpath='mode={.volumeBindingMode} provisioner={.provisioner}{"\n"}'

# WaitForFirstConsumer: unbound-during-scheduling is normal — check the pod side
kubectl describe pod -l app=postgres -n staging | grep -A6 'Events:' | tail -8

# Immediate: unbound means real failure — check provisioner events
kubectl get events -n staging --sort-by=.lastTimestamp | grep -iE 'provision|volume|pvc' | tail -10

# Allowed topologies can strand WaitForFirstConsumer claims
kubectl get sc fast-ssd -o jsonpath='{range .allowedTopologies}{@}{"\n"}{end}'

Dynamic Provisioning: When the Clerk Never Shows

Dynamic provisioning needs a running provisioner: the CSI external-provisioner sidecar watching for claims in its class and calling the cloud API to carve volumes. If the driver isn't installed, the sidecar is crash-looping, IAM denies the CreateVolume call, or the cloud quota is exhausted, claims wait with ProvisioningFailed events — or no events if the class points at a provisioner name nothing serves. Reinstalling random drivers (the Wednesday response) fixes nothing when the installed one lacks permissions.

Debug the chain outside-in. The class's provisioner field names the expected driver; the driver namespace should show running controller pods; their logs carry the cloud-side verdict — AccessDenied, quota exceeded, invalid parameter — verbatim. IAM and quota dominate in cloud: a fresh cluster with an old node role, or a region that hit its volume cap, fails every claim identically. Cross-check by provisioning manually (a test claim) while tailing controller logs: the error appears within seconds.

Harden the path once it works. Pin CSI driver versions in cluster bootstrapping (not latest-at-install), grant least-privilege storage IAM with a documented policy, alert on provisioner pod restarts, and keep a test-claim job that binds and deletes weekly — a synthetic probe proving the whole chain end to end. Provisioning is a clerk, a phone line (API), and permission to order rooms; monitor all three and the lobby stays empty.

# 1. Which driver should serve this class?
kubectl get sc fast-ssd -o jsonpath='{.provisioner}{"\n"}'

# 2. Is its controller alive? (labels vary by driver — adjust)
kubectl get pods -n kube-system | grep -i -E 'csi|provisioner|ebs|snapshot'
kubectl get pods -n kube-system -l app=csi-provisioner -o wide 2>/dev/null

# 3. The cloud-side verdict lives in controller logs
kubectl logs -n kube-system -l app=csi-provisioner --tail=80 2>/dev/null | grep -iE 'error|fail|denied|quota|invalid' | tail -10

# 4. Synthetic probe: a test claim while tailing logs
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: probe-bind-test
  namespace: default
spec:
  storageClassName: fast-ssd
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 1Gi
EOF
kubectl get pvc probe-bind-test -w --timeout=60s
kubectl delete pvc probe-bind-test

Bind It and Keep It Bound

Resolution order never changes: fix the named cause (class spelling, size/modes, provisioner health), then watch binding complete on its own — no pod restarts needed, since the scheduler notices the bound claim and places the waiting pod within seconds. Verify with kubectl get pvc showing Bound plus a volume name, then kubectl get pods showing the workload leaving Pending. If binding completes but pods still wait, the remaining constraint is elsewhere (node resources, affinity) — storage did its job.

The deployment manifest below shows the correct end state for the common dynamic case: claim and pod agreeing on name, namespace, and modes, against a class that exists. Note the claim lives in the same namespace as the pod — cross-namespace claims don't exist, and a typo'd namespace orphans the reference silently. Apply, watch Bound, watch Running: that's the whole ceremony when inputs are right.

Lock it with four guards. The CI class-existence gate from the incident. Standardized class names across environments. A Pending-pods alert (10 minutes, with describe output) so waits page before they become outages. And a weekly test-claim probe proving provisioning end to end. Storage binding is paperwork with a clerk — verify the names, keep the clerk healthy, and pods stop waiting in the lobby.

# Correct end state: claim + pod agreeing on name, namespace, modes.
# Apply, watch Bound, watch Running — no restarts needed.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-postgres
  namespace: staging
spec:
  storageClassName: fast-ssd   # must exist: kubectl get sc
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: staging           # same namespace as the claim
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data-postgres