Kubernetes Persistent Volume Claims: Dynamic Provisioning That Won't Fail at 3 AM

You've seen it: a pod stuck in Pending for hours because someone forgot to pre-create a PV. Or worse, a statefulset that silently loses data when a node dies because the volume was on local SSD with no replication. Static provisioning is a ticking time bomb. Dynamic provisioning is the fix — but only if you understand the trade-offs. By the end of this, you'll design StorageClasses that survive node failures, cloud outages, and your own mistakes.

Why Static Provisioning Is a Production Liability

Before dynamic provisioning, you had to create PVs manually — one per claim. That meant either over-provisioning (wasting cloud spend) or under-provisioning (blocking deployments). In a CI/CD pipeline, every new environment required a human to run kubectl create pv or a script that often failed because the storage backend was out of quota. Dynamic provisioning solves this by making the cluster request storage on demand, just like pods request CPU. The key enabler is the StorageClass, which tells Kubernetes which provisioner to call and what parameters to pass (e.g., disk type, IOPS, replication).

# io.thecodeforge — DevOps tutorial
# StorageClass for AWS EBS gp3 volumes with dynamic provisioning
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com  # AWS EBS CSI driver
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
reclaimPolicy: Delete  # Auto-delete PV and volume when PVC is deleted
volumeBindingMode: WaitForFirstConsumer  # Delay PV creation until pod is scheduled
allowVolumeExpansion: true

Designing a StorageClass That Matches Your Workload

Not all storage is equal. A MySQL database needs high IOPS and low latency. A log aggregator needs cheap, large capacity. A shared filesystem needs ReadWriteMany. Your StorageClass should encode these requirements. The parameters field is provisioner-specific — read the docs. For AWS EBS, you can set type (gp3, io1, st1), iops, throughput, and encrypted. For GCE PD, you set type (pd-ssd, pd-standard) and replication-type. For NFS, you set server and path. The volumeBindingMode matters: Immediate creates the PV as soon as the PVC is submitted, even if no pod needs it yet. WaitForFirstConsumer delays creation until a pod is scheduled, which respects topology constraints (e.g., zone affinity). Use WaitForFirstConsumer for stateful workloads to avoid orphaned volumes in wrong zones.

# io.thecodeforge — DevOps tutorial
# PVC for MySQL database using fast-ssd StorageClass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data
spec:
  accessModes:
    - ReadWriteOnce  # Only one pod can write at a time
  volumeMode: Filesystem
  resources:
    requests:
      storage: 100Gi
  storageClassName: fast-ssd  # Must match StorageClass name

Reclaim Policies: Delete vs Retain — Choose Wisely

The reclaimPolicy determines what happens to the PV and underlying storage when the PVC is deleted. Delete removes both — your cloud bill stays clean. Retain leaves the PV in Released state and the storage volume intact. You must manually delete the PV and then the cloud resource. Why would you ever use Retain? When the data is irreplaceable — a production database that must survive accidental PVC deletion. But Retain means you're on the hook for cleanup. I've seen teams burn thousands of dollars on orphaned EBS volumes because they forgot to clean up Retain PVs. My rule: use Delete for ephemeral environments and stateless apps. Use Retain only for stateful workloads with a backup strategy. And always set a StorageClass with Delete as default.

# io.thecodeforge — DevOps tutorial
# StorageClass with Retain policy for critical data
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: critical-data
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Retain  # PV and volume survive PVC deletion
volumeBindingMode: WaitForFirstConsumer

Volume Expansion: Growing Without Downtime

You will run out of disk space. It's not if, but when. Dynamic provisioning supports volume expansion if the StorageClass has allowVolumeExpansion: true and the CSI driver supports it. To expand, edit the PVC's spec.resources.requests.storage to a larger value. Kubernetes will call the provisioner to resize the underlying volume. The pod must be using the volume — if the filesystem supports online resize (most do), the new space is available immediately. If not, you may need to restart the pod. Always test expansion in staging. I've seen a resized volume fail because the underlying storage backend had a quota limit. Monitor kubectl describe pvc for events like Waiting for user to (re-)start a pod to finish file system resize of the volume.

# io.thecodeforge — DevOps tutorial
# Expand PVC from 100Gi to 200Gi
kubectl patch pvc mysql-data -p '{"spec": {"resources": {"requests": {"storage": "200Gi"}}}}'
# Watch resize progress
kubectl get pvc mysql-data -w

Topology Constraints: Keeping Volumes Close to Pods

Cloud storage volumes are zonal — an EBS volume in us-east-1a cannot be attached to a pod in us-east-1b. If your pod gets scheduled in a different zone, the volume will fail to attach. The fix is volumeBindingMode: WaitForFirstConsumer combined with allowedTopologies in the StorageClass. This tells Kubernetes to delay PV creation until a pod is scheduled, then create the PV in the same zone as the pod. Without this, you risk Multi-Attach errors or scheduling failures. For multi-zone clusters, always use WaitForFirstConsumer for stateful workloads. For stateless workloads, use Immediate and accept that the PV may be in a different zone — but then your pod must be scheduled in that zone (use nodeSelector or affinity).

# io.thecodeforge — DevOps tutorial
# StorageClass with zone-aware provisioning
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: zone-aware-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: topology.ebs.csi.aws.com/zone
    values:
    - us-east-1a
    - us-east-1b

When Dynamic Provisioning Is Overkill

Dynamic provisioning adds complexity: you need a CSI driver, IAM permissions, and StorageClass definitions. For a single-node cluster (e.g., Minikube, Docker Desktop), static provisioning with hostPath is simpler and faster. For development environments where data loss is acceptable, use emptyDir instead of PVCs. For production, dynamic provisioning is almost always the right choice — but don't use it for temporary scratch space. Also, if your storage backend doesn't support dynamic provisioning (e.g., some on-premise NFS servers), you're stuck with static PVs. In that case, use a manual PV with a nodeSelector to pin pods to the node with the NFS mount.

# io.thecodeforge — DevOps tutorial
# Static PV for NFS when dynamic provisioning is not available
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.1.100
    path: /exports/data
  persistentVolumeReclaimPolicy: Retain

Common Mistakes That Burn Production

I've seen these mistakes repeatedly. First: forgetting to set storageClassName in the PVC, which uses the cluster's default StorageClass — often the slow, generic one. Always specify it explicitly. Second: using ReadWriteOnce for a deployment with multiple replicas — only one pod can write at a time, others will fail. Use ReadWriteMany only if the backend supports it (NFS, EFS, Azure Files). Third: not setting volumeMode: Filesystem (the default) when you need a raw block device. For databases like Cassandra, raw block mode gives better performance. Fourth: ignoring volumeBindingMode — using Immediate in a multi-zone cluster causes cross-zone attachment failures. Fifth: not monitoring PVC capacity — a full volume causes pod evictions. Set up alerts on kubelet_volume_stats_used_bytes.

# io.thecodeforge — DevOps tutorial
# PVC for raw block volume (no filesystem)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cassandra-data
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block  # Raw block device, not filesystem
  resources:
    requests:
      storage: 50Gi
  storageClassName: fast-ssd