Kubernetes Persistent Volumes and Storage Classes: Stop Losing State When Pods Die

Every Kubernetes newbie learns the hard way: pods are ephemeral, but your database isn't. I've seen a startup lose 3 days of user data because they assumed a StatefulSet's PVC would survive a node failure. It didn't. PersistentVolumes and StorageClasses are how you tell Kubernetes 'this data matters.' Without them, you're gambling with state.

The problem is simple: containers crash, nodes fail, clusters scale down. Without persistent storage, your Postgres data vanishes when the pod restarts. The hack people used was hostPath—binding to a directory on the node. That works until your pod reschedules to a different node. Then you're staring at an empty directory and a corrupted database.

By the end of this, you'll be able to configure dynamic provisioning with a StorageClass, set reclaim policies that prevent accidental data deletion, and debug the three most common PV failures I've seen in production. You'll also know when to skip PVs entirely and use a managed database instead.

Why Static Provisioning Will Burn You

Before StorageClasses, you had to create PVs manually. Every developer who needed storage had to pester the ops team: 'Hey, can you provision a 50GB EBS volume and create a PV for it?' That's slow, error-prone, and doesn't scale. Worse, when the PV was full, you couldn't resize it without downtime. I've seen a team delete a production PV by accident because they forgot to set persistentVolumeReclaimPolicy: Retain. The PVC was deleted, and boom—the PV was recycled. Data gone.

Dynamic provisioning via StorageClass automates this. You define a StorageClass that points to a provisioner (e.g., kubernetes.io/aws-ebs), and when a PVC is created, Kubernetes calls the cloud provider's API to create the volume and bind it automatically. No manual steps. No ops bottleneck.

But here's the catch: not all StorageClasses are created equal. Some provisioners (like kubernetes.io/host-path or local) don't support dynamic provisioning at all. Always check the provisioner's documentation. For production, use cloud-native provisioners (EBS, GCE PD, Azure Disk) or a CSI driver like Rook/Ceph.

// io.thecodeforge — DevOps tutorial

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  fsType: ext4
  encrypted: "true"
reclaimPolicy: Retain  # Don't delete volume when PVC is deleted
volumeBindingMode: WaitForFirstConsumer  # Delay provisioning until pod scheduled
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: fast-ssd

Access Modes: ReadWriteOnce vs ReadOnlyMany vs ReadWriteMany

Access modes define how many pods can mount the volume and with what permissions. ReadWriteOnce (RWO) is the most common—one pod can read and write. ReadOnlyMany (ROX) allows many pods to read simultaneously. ReadWriteMany (RWX) allows many pods to read and write—but this is the trickiest.

RWX requires a filesystem that supports concurrent writes, like NFS or GlusterFS. Cloud block storage (EBS, GCE PD) only supports RWO. I've seen teams try to use an EBS volume with RWX and wonder why their pods crash with mount.nfs: access denied by server while mounting. The fix is to use a CSI driver that supports RWX, like Amazon EFS or Azure Files.

For stateful workloads like databases, use RWO. For shared configuration or logs, use ROX or RWX with a proper distributed filesystem. Never assume RWX works—test it first.

// io.thecodeforge — DevOps tutorial

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-logs
spec:
  accessModes:
    - ReadWriteMany  # Requires a volume that supports RWX (e.g., NFS, EFS)
  resources:
    requests:
      storage: 10Gi
  storageClassName: efs-sc

Reclaim Policies: Retain, Delete, or Recycle

When a PVC is deleted, what happens to the PV? That's the reclaim policy. Delete removes the PV and the underlying storage asset. Retain keeps the PV and volume—you must manually delete them. Recycle (deprecated) scrubs the volume and makes it available again.

In production, always use Retain for any volume with data you care about. I've seen a junior engineer delete a PVC thinking it would just unbind the PV—but the policy was Delete, and the entire EBS volume was gone. No recovery. The fix is to set persistentVolumeReclaimPolicy: Retain on the StorageClass or on individual PVs.

For ephemeral storage (e.g., build artifacts), Delete is fine. But for databases, queues, or any stateful service, Retain is non-negotiable. And always have a backup strategy—Retain doesn't protect against accidental deletion of the PV itself.

// io.thecodeforge — DevOps tutorial

apiVersion: v1
kind: PersistentVolume
metadata:
  name: manual-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain  # Volume survives PVC deletion
  awsElasticBlockStore:
    volumeID: vol-0a1b2c3d4e5f67890
    fsType: ext4

Volume Binding Modes: Immediate vs WaitForFirstConsumer

Volume binding mode controls when the PV is provisioned and bound to the PVC. Immediate creates the volume as soon as the PVC is created. WaitForFirstConsumer delays provisioning until a pod using the PVC is scheduled.

Why use WaitForFirstConsumer? In multi-zone clusters, Immediate might provision a volume in zone A, but the pod gets scheduled in zone B. Then the pod can't mount the volume because it's in a different availability zone. I've seen this bring down a multi-region deployment—pods were stuck in ContainerCreating with volume zone mismatch errors.

WaitForFirstConsumer ensures the volume is provisioned in the same zone as the pod. This is critical for cloud providers where volumes are zonal (EBS, GCE PD). For regional volumes (like Azure Disk), Immediate is fine. Always set volumeBindingMode: WaitForFirstConsumer for zonal storage.

// io.thecodeforge — DevOps tutorial

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: zonal-ssd
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
  zones: us-central1-a,us-central1-b
volumeBindingMode: WaitForFirstConsumer  # Provision in same zone as pod
reclaimPolicy: Retain

Expanding Volumes Without Downtime

You deployed a 50GB Postgres instance, and now it's at 48GB. You need more space. With static provisioning, you'd have to create a new PV, migrate data, and update the PVC—downtime guaranteed. With dynamic provisioning and allowVolumeExpansion: true, you can edit the PVC's storage request and Kubernetes will expand the volume in place.

But there are caveats. Not all volume types support expansion. EBS gp2/gp3 does, but magnetic (standard) doesn't. Also, the filesystem must support resizing—ext4 and xfs do, but not all CSI drivers handle it automatically. I've seen a team expand a PVC, but the pod still saw the old size because the filesystem wasn't resized. The fix was to run resize2fs inside the pod (or use a CSI driver that does it automatically).

To enable expansion, set allowVolumeExpansion: true in the StorageClass. Then edit the PVC: kubectl edit pvc <name> and change spec.resources.requests.storage. The volume will expand, but the pod may need a restart to see the new size.

// io.thecodeforge — DevOps tutorial

# Step 1: Enable expansion in StorageClass
kubectl patch storageclass standard -p '{"allowVolumeExpansion": true}'

# Step 2: Increase PVC size
kubectl patch pvc mysql-data -p '{"spec": {"resources": {"requests": {"storage": "100Gi"}}}}'

# Step 3: Verify expansion
kubectl get pvc mysql-data -o jsonpath='{.status.capacity.storage}'
# Output: 100Gi

# Step 4: If filesystem not resized, exec into pod and run:
kubectl exec mysql-0 -- resize2fs /var/lib/mysql

When Not to Use Persistent Volumes

PVs are great for stateful workloads, but they're not the answer to every storage problem. If you're running a stateless application, don't use PVs—use emptyDir or ConfigMap. If you need a database, consider a managed service (RDS, Cloud SQL) instead of running it yourself on PVs. Managed databases handle backups, replication, and failover—you don't have to.

I've seen teams run MySQL on PVs with a single replica, no backup strategy, and wonder why they lost data when the node died. For production databases, use a managed service or at least a StatefulSet with regular backups. PVs are infrastructure, not a backup solution.

Also, avoid PVs for temporary or cache data. Use emptyDir with a memory-backed medium (if your node has enough RAM) or a sidecar with a caching layer like Redis. PVs add latency and complexity—only use them when data must survive pod restarts.

// io.thecodeforge — DevOps tutorial

apiVersion: v1
kind: Pod
metadata:
  name: cache-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - mountPath: /cache
      name: temp-storage
  volumes:
  - name: temp-storage
    emptyDir:
      medium: Memory  # Use RAM for speed
      sizeLimit: 1Gi

Debugging PV Failures: The Three Most Common Issues

1. PVC stuck in Pending: Run kubectl describe pvc <name>. Look for events like no persistent volumes available for this claim and no storage class is set. Fix: add storageClassName to PVC or create a default StorageClass.
2. Pod stuck in ContainerCreating: Run kubectl describe pod <name>. Look for volume node affinity conflict or multi-attach error. Fix: ensure volume is in the same zone as the pod (use WaitForFirstConsumer), or delete the pod that's holding the volume (if multi-attach not supported).
3. Volume mount fails with permission denied: Check the filesystem permissions. Many PVs (like NFS) mount as root. If your pod runs as non-root, you need to set fsGroup in the pod's security context. I've seen this with Jenkins agents—they couldn't write to the workspace because the PV was owned by root.

// io.thecodeforge — DevOps tutorial

# Check PVC status and events
kubectl describe pvc mysql-data

# Check pod events for volume issues
kubectl describe pod mysql-0

# Check if StorageClass exists
kubectl get storageclass

# Check PV details
kubectl get pv
kubectl describe pv <pv-name>

# For permission issues, add fsGroup to pod spec:
kubectl patch deployment mysql -p '{"spec":{"template":{"spec":{"securityContext":{"fsGroup":1000}}}}}'