Kubernetes Volumes and Storage — PV, PVC, StorageClass

Welcome to Kubernetes Volumes and Storage — PV, PVC, StorageClass. We'll break this down from first principles.

Why Kubernetes Storage Is Different

In traditional VM deployments, storage is typically attached to a specific host and survives reboots. Containers, by design, are ephemeral — their filesystem disappears when the pod dies. This creates a fundamental tension: stateful applications (databases, message queues, file servers) need persistent data, but Kubernetes treats pods as cattle. The solution is a storage abstraction layer that decouples the lifecycle of data from the lifecycle of pods. Kubernetes volumes are not just directories; they are API objects with defined lifecycles, access modes, and reclaim policies. Understanding this paradigm shift is critical before diving into PVs and PVCs. Without it, you'll end up with data loss, orphaned volumes, or cloud bills from unattached disks.

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: writer
    image: busybox
    command: ["sh", "-c", "echo 'data' > /data/file && sleep 3600"]
    volumeMounts:
    - name: scratch
      mountPath: /data
  volumes:
  - name: scratch
    emptyDir: {}

PersistentVolume: The Cluster Storage Resource

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned via StorageClass. It is a cluster resource, just like a node, and exists independently of any pod. PVs have a lifecycle: Available, Bound, Released, and Failed. They define capacity, access modes (ReadWriteOnce, ReadOnlyMany, ReadWriteMany), reclaim policy (Retain, Delete, Recycle), and storage class. In production, you rarely create PVs manually — you define a StorageClass and let the cluster provision them on demand. However, understanding PVs is essential for debugging binding issues, capacity planning, and disaster recovery. A PV can be backed by NFS, iSCSI, cloud disks (EBS, GCE PD), or local SSD.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-manual
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /mnt/data

PersistentVolumeClaim: Requesting Storage

A PersistentVolumeClaim (PVC) is a request for storage by a user. It specifies size, access mode, and optionally a storage class. Kubernetes matches a PVC to a PV that satisfies the request. If no matching PV exists and a StorageClass is specified, dynamic provisioning kicks in. PVCs are namespaced, so different teams can request storage without stepping on each other. Once bound, a PVC is mounted into a pod as a volume. The binding is exclusive: a PV can only be bound to one PVC at a time (unless using ReadWriteMany). PVCs also support label selectors for more granular matching. In production, you should always specify a storageClassName to avoid accidentally binding to the wrong PV.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: fast

StorageClass: Dynamic Provisioning

StorageClass is the magic that makes Kubernetes storage truly cloud-native. It defines a provisioner (e.g., kubernetes.io/aws-ebs, csi-hostpath), parameters (type, IOPS, replication), and reclaim policy. When a PVC requests a StorageClass, the provisioner automatically creates a PV and binds it. This eliminates manual PV creation and enables self-service storage for developers. StorageClasses also support volume binding modes (Immediate vs WaitForFirstConsumer) and allow policies. In production, you should have at least two StorageClasses: one for fast SSD-backed volumes (databases) and one for cheaper HDD-backed volumes (logs, backups). Always test the provisioner's behavior with your cloud provider — some have quirks like volume limits per instance type.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iopsPerGB: "10"
  fsType: ext4
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Access Modes and Their Real-World Implications

Access modes define how many nodes can mount the volume and with what permissions. ReadWriteOnce (RWO) allows a single node to mount read-write. ReadOnlyMany (ROX) allows many nodes to mount read-only. ReadWriteMany (RWX) allows many nodes to mount read-write. Not all storage backends support all modes. For example, EBS only supports RWO; NFS supports RWX. Choosing the wrong access mode leads to pod startup failures or data corruption. In production, use RWO for databases (single writer), ROX for config maps or static data, and RWX for shared file systems (e.g., WordPress uploads). RWX is notoriously tricky — ensure your application handles concurrent writes correctly.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-storage
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: nfs

Reclaim Policies: What Happens When You Delete a PVC

Reclaim policy determines what happens to a PV when its bound PVC is deleted. Retain keeps the PV and its data, but it becomes Released and cannot be reused until manually reclaimed. Delete removes both the PV and the underlying storage asset (e.g., EBS volume). Recycle (deprecated) scrubs the volume and makes it available again. In production, always use Retain for stateful workloads (databases, message queues) to prevent accidental data loss. Use Delete for ephemeral storage (CI artifacts, temp data). Never use Recycle — it's deprecated and insecure. Implement a backup strategy for Retain volumes; they can become orphaned and cost money if not cleaned up.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-retain
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /mnt/data

Volume Snapshots and Cloning

Kubernetes supports volume snapshots and cloning via CSI drivers. Snapshots capture a point-in-time copy of a volume, useful for backups and disaster recovery. Cloning creates a new volume that is a duplicate of an existing one, useful for creating dev databases from production snapshots. Both are API objects: VolumeSnapshot, VolumeSnapshotContent, VolumeSnapshotClass. Not all CSI drivers support these features. In production, use snapshots for scheduled backups (e.g., via Velero) and clones for rapid environment provisioning. Be aware of performance impact: taking a snapshot of a heavily written volume can cause latency spikes.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: db-snapshot-20260712
spec:
  volumeSnapshotClassName: csi-aws-vsc
  source:
    persistentVolumeClaimName: postgres-pvc

CSI Drivers: The Modern Storage Interface

Container Storage Interface (CSI) is the standard for exposing storage systems to Kubernetes. CSI drivers run as DaemonSets and sidecars, handling volume operations (create, delete, mount, unmount). They replace in-tree volume plugins (e.g., kubernetes.io/aws-ebs) which are being deprecated. CSI drivers support features like snapshots, resize, topology, and inline volumes. In production, always use CSI drivers for cloud storage (EBS, GCE PD, Azure Disk) and third-party storage (Portworx, Rook/Ceph). They are more reliable, feature-rich, and maintained by the vendor. Migrate from in-tree plugins before they are removed in Kubernetes 1.28+.

# Example: Installing AWS EBS CSI driver
# Using Helm
helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
helm upgrade --install aws-ebs-csi-driver \
  --namespace kube-system \
  --set controller.serviceAccount.create=true \
  aws-ebs-csi-driver/aws-ebs-csi-driver

StatefulSets and Storage: The Right Way to Run Databases

StatefulSets are the workload API for stateful applications. They provide stable network identities, ordered deployment/scaling, and persistent storage via volumeClaimTemplates. Each replica gets its own PVC, ensuring data isolation. This is ideal for databases like Cassandra, MySQL, or Kafka. However, StatefulSets are not a silver bullet — they require careful planning for backup, recovery, and scaling. In production, use volumeClaimTemplates with a dedicated StorageClass (e.g., fast SSD). Be aware that scaling down a StatefulSet does not delete PVCs by default — you must manually clean up or use a custom controller.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast
      resources:
        requests:
          storage: 100Gi

Ephemeral Volumes and Inline CSI

Kubernetes 1.22+ introduced generic ephemeral volumes, which combine the lifecycle of an emptyDir with the features of a PVC (snapshots, resize, storage class). They are created inline in a pod spec and deleted when the pod is removed. This is useful for scratch space that needs more capacity or performance than emptyDir. Inline CSI volumes allow you to directly reference a CSI volume in a pod spec without a PVC. In production, use ephemeral volumes for CI runners, batch jobs, or any workload where data doesn't need to outlive the pod. They reduce management overhead but still incur storage costs.

apiVersion: v1
kind: Pod
metadata:
  name: ephemeral-pod
spec:
  containers:
  - name: app
    image: busybox
    command: ["sh", "-c", "dd if=/dev/zero of=/data/largefile bs=1M count=1024 && sleep 3600"]
    volumeMounts:
    - name: fast-scratch
      mountPath: /data
  volumes:
  - name: fast-scratch
    ephemeral:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          storageClassName: fast
          resources:
            requests:
              storage: 10Gi

Troubleshooting Storage Issues

Storage problems are among the hardest to debug in Kubernetes. Common issues: PVC stuck in Pending (no matching PV or StorageClass), pod stuck in ContainerCreating (volume mount failure), or volume not detaching (node issues). Start by describing the PVC and PV: kubectl describe pvc <name> shows events. Check the CSI driver logs (kubectl logs -n kube-system <csi-controller-pod>). Verify node conditions and kubelet logs. For cloud volumes, check the cloud console for volume state. In production, set up monitoring for volume metrics (disk usage, IOPS, latency) and alerts for PVC binding failures. Always have a rollback plan for storage changes.

# Check PVC status
kubectl describe pvc my-pvc

# Check PV status
kubectl get pv

# Check events in namespace
kubectl get events --sort-by='.lastTimestamp'

# Check CSI driver pods
kubectl get pods -n kube-system -l app=ebs-csi-controller

# Check kubelet logs for mount errors
journalctl -u kubelet -n 50 | grep -i volume

Production Best Practices for Kubernetes Storage

After years of running stateful workloads in production, here are the non-negotiables: 1) Use CSI drivers for all storage. 2) Define at least two StorageClasses (fast and slow). 3) Set reclaimPolicy: Retain for all production PVCs. 4) Use volumeBindingMode: WaitForFirstConsumer for cloud volumes. 5) Implement backup and disaster recovery with Velero or similar. 6) Monitor volume usage and set alerts for >80% capacity. 7) Use PodDisruptionBudgets with StatefulSets to prevent data loss during node drains. 8) Test storage failover scenarios regularly. 9) Avoid hostPath and emptyDir for anything important. 10) Document your storage topology and recovery procedures.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: mysql-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: mysql