Kubernetes StatefulSets — Kafka's 14-Hour Split-Brain

Stateful applications in Kubernetes demand persistent identity, stable network names, and ordered lifecycle—none of which vanilla Deployments provide. Without StatefulSets, your database pods will fight over the same volumes, restart in random order, and break quorum. StatefulSets enforce pod-ordering and bind storage to identity, letting you run Postgres, Cassandra, or Kafka the same way you run stateless microservices, without data corruption or split-brain disasters.

What is a Kubernetes StatefulSet?

A StatefulSet is a Kubernetes workload API object designed specifically for managing stateful applications. The name is deliberately chosen to contrast with the default Kubernetes mental model — stateless pods that are interchangeable, ephemeral, and replaceable. StatefulSets break that model intentionally and provide three guarantees that distributed stateful systems depend on at the protocol level.

First: stable, unique network identity. Each pod in a StatefulSet receives a deterministic name based on its ordinal index — web-0, web-1, web-2. This name is not a random hash suffix. It survives pod restarts, node failures, and rescheduling. Combined with a headless Service, this creates a stable DNS entry that other pods can use to discover and connect to a specific instance — which is critical for systems like ZooKeeper, etcd, and Kafka where cluster membership is a first-class concept baked into the protocol.

Second: stable, persistent storage. volumeClaimTemplates provision a dedicated PersistentVolumeClaim per pod using a naming convention tied to the pod's ordinal. When a pod is deleted and recreated, it reattaches to the same PVC. The storage follows the identity, not the node.

Third: ordered deployment and scaling. Pods are created sequentially — pod 0 must be Running and Ready before pod 1 is created. Pods are deleted in reverse order — pod N-1 before pod N-2 before pod 0. This ordering is not a soft preference — it is enforced by the StatefulSet controller at the API level.

Understanding which of these three guarantees your workload actually needs is more important than knowing the YAML syntax. A single-instance PostgreSQL database needs storage persistence but does not need the ordering guarantees. A ZooKeeper ensemble needs all three.

# The headless Service must be created BEFORE the StatefulSet.
# The StatefulSet controller validates the serviceName reference on creation.
# Without the headless Service, per-pod DNS records cannot be created.
apiVersion: v1
kind: Service
metadata:
  name: io-thecodeforge-headless
  labels:
    app: io-thecodeforge-db
spec:
  clusterIP: None  # This is what makes it headless — no virtual IP, direct pod DNS records
  selector:
    app: io-thecodeforge-db  # Must match the StatefulSet pod labels exactly
  ports:
    - port: 5432
      name: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: io-thecodeforge-db
spec:
  serviceName: io-thecodeforge-headless  # References the headless Service above
  replicas: 3
  selector:
    matchLabels:
      app: io-thecodeforge-db
  template:
    metadata:
      labels:
        app: io-thecodeforge-db
    spec:
      terminationGracePeriodSeconds: 120  # Give PostgreSQL time to flush WAL before SIGKILL
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: io-thecodeforge-db-secret
                  key: password
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "postgres"]
            initialDelaySeconds: 10
            periodSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  # volumeClaimTemplates creates one PVC per pod:
  # data-io-thecodeforge-db-0, data-io-thecodeforge-db-1, data-io-thecodeforge-db-2
  # These PVCs survive pod deletion and are reattached when pods are recreated.
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]  # One node at a time — appropriate for databases
        storageClassName: io-thecodeforge-ssd
        resources:
          requests:
            storage: 50Gi

Ordered Operations: Creation, Deletion, and Rolling Updates

StatefulSet operations are strictly ordered in ways that feel unusual if you are used to Deployment behaviour. Understanding this ordering is essential for both operating StatefulSets correctly and for debugging when something gets stuck.

Pod creation is strictly sequential: the controller creates pod 0, then waits until it is Running and Ready before creating pod 1. Pod 1 must be Running and Ready before pod 2 is created. This is not just about startup — it reflects the dependency structure of many distributed systems where node 0 bootstraps the cluster and node 1 joins as a follower. If you scale a StatefulSet from 3 to 5 replicas, pods 3 and 4 are created sequentially in that order.

Pod deletion is strictly reverse-sequential: the controller deletes pod N-1, waits for it to fully terminate, then deletes pod N-2, and so on to pod 0. This preserves quorum during scale-down — a ZooKeeper ensemble being scaled from 5 to 3 nodes loses its two highest-ordinal nodes first, maintaining the 3-node quorum throughout the process rather than potentially losing the primary.

Rolling updates follow a special reverse-ordinal pattern that is different from what most engineers expect. The controller updates pod N-1 first, waits for it to become Ready, then proceeds to pod N-2, down to pod 0 last. Pod 0, which often holds the most critical role in distributed systems (initial voter in ZooKeeper, partition leader for critical topics in Kafka), is updated last — preserving cluster stability for as long as possible during the rollout.

The partition field in the rolling update strategy is one of the most useful and most forgotten StatefulSet features. Setting partition: 3 in a 5-pod StatefulSet means only pods 3 and 4 update when you apply a new template. Pods 0, 1, and 2 remain on the old version. This is a native canary mechanism built into the StatefulSet API — you validate pods 3 and 4 under production traffic, then lower partition to 0 to complete the rollout. The failure mode to remember: if you set partition for a canary and forget to reset it to 0, your cluster runs in a permanently split-version state indefinitely.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: io-thecodeforge-db
spec:
  serviceName: io-thecodeforge-headless
  replicas: 5
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      # Only pods with ordinal >= 3 will be updated when the template changes.
      # Pods 0, 1, and 2 remain on the current version.
      # This is the canary pattern: validate pods 3 and 4 first.
      # To complete the rollout: kubectl patch statefulset io-thecodeforge-db
      #   --type=merge -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'
      partition: 3
  selector:
    matchLabels:
      app: io-thecodeforge-db
  template:
    metadata:
      labels:
        app: io-thecodeforge-db
    spec:
      # 90 seconds gives PostgreSQL enough time to:
      # 1. Finish in-progress transactions
      # 2. Flush dirty buffers to disk
      # 3. Write a clean checkpoint to WAL
      # 4. Cleanly release file locks
      # Without sufficient grace period, the next startup triggers crash recovery.
      terminationGracePeriodSeconds: 90
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "postgres"]
            initialDelaySeconds: 10
            periodSeconds: 5
            # failureThreshold * periodSeconds = how long before the rollout is blocked
            # 3 * 5 = 15 seconds of failed readiness before the controller stops advancing
            failureThreshold: 3
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: io-thecodeforge-ssd
        resources:
          requests:
            storage: 50Gi

# Monitor rollout progress:
# kubectl rollout status statefulset/io-thecodeforge-db
# kubectl get statefulset io-thecodeforge-db -o jsonpath=\
#   '{.status.currentRevision} {.status.updateRevision}'

PVC Lifecycle: Storage That Follows Identity

StatefulSets use volumeClaimTemplates to provision one PersistentVolumeClaim per pod. The naming convention is deterministic: template-name-statefulset-name-ordinal. For a StatefulSet named io-thecodeforge-db with a template named data, the PVCs are data-io-thecodeforge-db-0, data-io-thecodeforge-db-1, and data-io-thecodeforge-db-2. This naming is not configurable — it is generated by the StatefulSet controller.

The storage lifecycle has two properties that every engineer running StatefulSets in production must understand deeply.

First, PVCs survive pod deletion. When a pod is deleted — whether by a rolling update, a manual kubectl delete pod, a node failure, or a scale-down — its PVC is not deleted. When the pod is recreated with the same ordinal, it reattaches to the same PVC. This is the 'sticky storage' guarantee. It is what makes your PostgreSQL data directory survive node failures without data loss.

Second, PVCs are NOT deleted when the StatefulSet is deleted. This is a deliberate safety mechanism — accidentally deleting a StatefulSet should not destroy production databases. But it means that scaling down a StatefulSet from 5 to 3 replicas leaves two orphaned PVCs (data-name-3 and data-name-4) that consume cloud storage indefinitely. At $0.08-0.15 per GB per month on most cloud providers, a 1TB database with ten orphaned scale-down PVCs accumulates $800-1500 per month in silent storage waste. This is one of the most common cost anomalies in Kubernetes clusters and one of the least visible.

Kubernetes 1.27 introduced the persistentVolumeClaimRetentionPolicy field on StatefulSets, which allows you to configure automatic PVC deletion on scale-down (whenScaled) or StatefulSet deletion (whenDeleted). For most production stateful workloads, setting whenScaled to Delete is appropriate — the PVC for a scaled-down pod can reasonably be considered ephemeral. Setting whenDeleted to Delete is more dangerous and should only be used when you have confirmed out-of-cluster backups.

# Kubernetes 1.27+ introduces persistentVolumeClaimRetentionPolicy
# This controls what happens to PVCs when pods are deleted or the StatefulSet scales down.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: io-thecodeforge-db
spec:
  serviceName: io-thecodeforge-headless
  replicas: 3
  # Available from Kubernetes 1.27+ (stable in 1.29+)
  persistentVolumeClaimRetentionPolicy:
    # whenScaled: what happens to PVCs when replicas is reduced
    #   Delete  - PVC is deleted when its pod is scaled away (use with confirmed backups)
    #   Retain  - PVC survives scale-down (default, safe but causes orphaned volume drift)
    whenScaled: Retain
    # whenDeleted: what happens to PVCs when the StatefulSet is deleted
    #   Delete  - PVCs are deleted with the StatefulSet (DANGEROUS — only with backups)
    #   Retain  - PVCs survive StatefulSet deletion (default, safe)
    whenDeleted: Retain
  selector:
    matchLabels:
      app: io-thecodeforge-db
  template:
    metadata:
      labels:
        app: io-thecodeforge-db
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: io-thecodeforge-ssd
        resources:
          requests:
            storage: 50Gi

---
# Manual PVC lifecycle management for clusters on Kubernetes < 1.27:

# After scaling down from 5 to 3 replicas, list orphaned PVCs:
# kubectl get pvc -l app=io-thecodeforge-db
# NAME                           STATUS  CAPACITY
# data-io-thecodeforge-db-0      Bound   50Gi   <- still in use
# data-io-thecodeforge-db-1      Bound   50Gi   <- still in use
# data-io-thecodeforge-db-2      Bound   50Gi   <- still in use
# data-io-thecodeforge-db-3      Bound   50Gi   <- ORPHANED (pod deleted)
# data-io-thecodeforge-db-4      Bound   50Gi   <- ORPHANED (pod deleted)

# Clean up orphaned PVCs after confirming data is not needed:
# kubectl delete pvc data-io-thecodeforge-db-3 data-io-thecodeforge-db-4

# WARNING: PVC deletion is IRREVERSIBLE
# The underlying PersistentVolume and its data are gone permanently.
# Always confirm you have a recent backup before deleting any database PVC.

Headless Services and DNS: How Pods Find Each Other

A headless Service is the DNS backbone of a StatefulSet. Without it, StatefulSet pods have stable names but no way for other pods to resolve those names to IP addresses. With it, every pod in the StatefulSet gets an individual DNS A record that points directly to that pod's IP — bypassing the load-balancing layer that regular Services add.

The distinction between a regular Service and a headless Service is important to internalise. A regular Service (clusterIP: something) creates one DNS name that resolves to a virtual IP, and kube-proxy load-balances traffic from that virtual IP to any matching pod. You cannot target a specific pod by DNS with a regular Service. A headless Service (clusterIP: None) creates no virtual IP and no load balancing. Instead, CoreDNS creates individual A records for each pod, one per pod, each pointing to that pod's actual IP address. This is how pod-to-pod targeting works.

The DNS naming convention for StatefulSet pods is: pod-name.service-name.namespace.svc.cluster.local. For a StatefulSet named db with a headless Service named db-headless in namespace prod, pod 0's DNS entry is db-0.db-headless.prod.svc.cluster.local. This entry is stable — it always points to the pod with that identity, regardless of which node it runs on. When the pod is rescheduled to a different node with a different IP, CoreDNS updates the A record to reflect the new IP. The DNS name remains constant; only the IP it resolves to changes.

The headless Service also creates a SRV record for each pod: _port-name._protocol.service-name.namespace.svc.cluster.local. SRV records carry both the hostname and port, which is how distributed systems like Kafka and ZooKeeper bootstrap cluster membership at startup — they query the SRV record to discover all current pod hostnames without needing to know ordinal count in advance.

One practical detail: CoreDNS caches DNS responses with a TTL. For headless Services the default TTL is 5 seconds. If a pod is rescheduled quickly, there is a brief window where other pods may try to connect to the old IP before the cache expires. Design your application's connection retry logic to tolerate this — most database connection pools handle it correctly if configured with appropriate connection timeouts.

# The headless Service — created before the StatefulSet, referenced by serviceName field.
apiVersion: v1
kind: Service
metadata:
  name: io-thecodeforge-headless
  namespace: prod
  labels:
    app: io-thecodeforge-db
spec:
  clusterIP: None  # No virtual IP — each pod gets its own DNS A record
  publishNotReadyAddresses: false  # Only Ready pods get DNS records (default: false)
  # Set to true for stateful systems that need to discover all members
  # including those still initializing — useful for ZooKeeper ensemble bootstrap
  selector:
    app: io-thecodeforge-db  # Must match StatefulSet pod labels EXACTLY
  ports:
    - port: 5432
      targetPort: 5432
      name: postgres

# --- DNS Records Created by CoreDNS ---
#
# Per-pod A records (stable, direct — the primary use case):
#   io-thecodeforge-db-0.io-thecodeforge-headless.prod.svc.cluster.local -> 10.244.1.5
#   io-thecodeforge-db-1.io-thecodeforge-headless.prod.svc.cluster.local -> 10.244.2.8
#   io-thecodeforge-db-2.io-thecodeforge-headless.prod.svc.cluster.local -> 10.244.3.12
#
# SRV records (used for automatic cluster member discovery):
#   _postgres._tcp.io-thecodeforge-headless.prod.svc.cluster.local
#   -> SRV: 0 50 5432 io-thecodeforge-db-0.io-thecodeforge-headless.prod.svc.cluster.local
#   -> SRV: 0 50 5432 io-thecodeforge-db-1.io-thecodeforge-headless.prod.svc.cluster.local
#   -> SRV: 0 50 5432 io-thecodeforge-db-2.io-thecodeforge-headless.prod.svc.cluster.local
#
# Service-level A record (returns all pod IPs — round-robin, not load-balanced):
#   io-thecodeforge-headless.prod.svc.cluster.local -> 10.244.1.5, 10.244.2.8, 10.244.3.12

# --- Verify DNS resolution from inside the cluster ---
# kubectl exec io-thecodeforge-db-1 -- \n#   nslookup io-thecodeforge-db-0.io-thecodeforge-headless.prod.svc.cluster.local
# Expected output: Address: 10.244.1.5

PodDisruptionBudget: Protecting Quorum During Voluntary Disruptions

PodDisruptionBudgets (PDBs) are among the most important and most skipped Kubernetes objects for StatefulSets in production. A PDB is a policy object that limits how many pods in a set can be simultaneously unavailable during voluntary disruptions — node drains, cluster upgrades, autoscaler scale-downs, and manual evictions.

For a quorum-based system, the requirement is concrete: a 3-node ZooKeeper cluster needs at least 2 nodes to maintain quorum. A 5-node etcd cluster needs at least 3 nodes. A PDB with minAvailable: 2 tells Kubernetes — during any voluntary disruption, you must ensure at least 2 of my pods are running and Ready before proceeding with the eviction. If a node drain would cause a second pod to become unavailable while the first is still evicting, the drain blocks until the first pod is rescheduled and Ready elsewhere.

This blocking behaviour is the point. Without a PDB, kubectl drain proceeds freely, evicting pods without regard for quorum. A node drain during a cluster upgrade with three StatefulSet pods all on the same node (a common situation if pod anti-affinity is not configured) will evict all three simultaneously, destroying quorum completely.

The combination of PDB plus pod anti-affinity is the production-grade pattern. The PDB protects against voluntary disruptions. Pod anti-affinity with requiredDuringSchedulingIgnoredDuringExecution and topologyKey: kubernetes.io/hostname protects against involuntary disruptions by ensuring no two pods land on the same node. Together, they mean: node drains cannot break quorum, and a single node failure cannot take out more than one pod.

One operational detail: kubectl drain respects PDBs. kubectl delete pod does not. If your operational runbook uses kubectl delete pod to perform maintenance, PDBs provide no protection. Use kubectl drain or kubernetes eviction API calls for any maintenance that should respect disruption budgets.

# PodDisruptionBudget — protects quorum during voluntary disruptions
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: io-thecodeforge-db-pdb
  namespace: prod
spec:
  # For a 3-node cluster: minAvailable: 2 (majority = quorum)
  # For a 5-node cluster: minAvailable: 3 (majority = quorum)
  # For a 7-node cluster: minAvailable: 4 (majority = quorum)
  # Rule: minAvailable = floor(replicas / 2) + 1
  minAvailable: 2
  selector:
    matchLabels:
      app: io-thecodeforge-db

# Alternative using maxUnavailable (equivalent for a 3-pod set):
# spec:
#   maxUnavailable: 1
#   selector:
#     matchLabels:
#       app: io-thecodeforge-db

---
# Pod anti-affinity — spreads pods across nodes to protect against involuntary failures
# PDB protects against voluntary disruptions (drain, upgrade)
# Anti-affinity protects against involuntary disruptions (node crash)
# You need BOTH for production-grade StatefulSet resilience.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: io-thecodeforge-db
  namespace: prod
spec:
  serviceName: io-thecodeforge-headless
  replicas: 3
  selector:
    matchLabels:
      app: io-thecodeforge-db
  template:
    metadata:
      labels:
        app: io-thecodeforge-db
    spec:
      affinity:
        podAntiAffinity:
          # required = hard constraint: pods WILL NOT be scheduled on the same node
          # preferred = soft preference: Kubernetes tries but can violate if necessary
          # For quorum-sensitive systems, use required.
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - io-thecodeforge-db
              # kubernetes.io/hostname = spread across nodes
              # topology.kubernetes.io/zone = spread across AZs (stronger guarantee)
              topologyKey: topology.kubernetes.io/zone
      containers:
        - name: postgres
          image: postgres:16-alpine
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: io-thecodeforge-ssd
        resources:
          requests:
            storage: 50Gi

Before You Begin: The Prerequisites Nobody Tells You About

StatefulSets look simple until they break your database at 3 AM. Before you touch one, you need to understand three things that most tutorials gloss over.

First, your cluster needs a dynamic PersistentVolume provisioner with a StorageClass that supports ReadWriteOnce. If you're on a bare-metal cluster without a proper CSI driver, your PVCs will stay Pending forever. Check with kubectl get storageclass and verify your provisioner supports volumeBindingMode: WaitForFirstConsumer for StatefulSets with topology constraints.

Second, you need a headless Service. This isn't optional — StatefulSets rely on DNS records to assign stable hostnames like pod-name-0.service-name.namespace.svc.cluster.local. Without a headless Service (spec.clusterIP: None), your pods won't get predictable network identities.

Third, understand that StatefulSets require ordered pod management by default. This means scaling down waits for pod-2 to terminate before touching pod-1. If your app doesn't need ordering, you're paying a latency tax for nothing — consider setting spec.podManagementPolicy: Parallel.

// io.thecodeforge — devops tutorial

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Creating a StatefulSet: The Manifest That Actually Works

Stop copy-pasting random YAML from blogs. Here's how a real production StatefulSet looks for a PostgreSQL cluster — and why every field exists.

The critical parts: serviceName must match your headless Service's name exactly. Kubernetes uses this to generate DNS records. spec.selector.matchLabels must match the pod template labels — no mismatch or the StatefulSet controller rejects it silently.

Notice volumeClaimTemplates instead of volumes. This is the magic that gives each pod its own PVC. Each template creates one PVC per replica, named {volume-name}-{statefulset-name}-{ordinal}. The PVC survives pod deletion, so when the pod comes back (maybe on a different node), it reattaches to the same storage.

The podManagementPolicy defaults to OrderedReady, which waits for pod-0 to be Running and Ready before creating pod-1. This is required for quorum-based apps like databases. If you're running a queue worker or cache, switch to Parallel to avoid the serial bottleneck.

// io.thecodeforge — devops tutorial

apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
spec:
  clusterIP: None
  selector:
    app: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-cluster
spec:
  serviceName: postgres-headless
  replicas: 3
  podManagementPolicy: OrderedReady
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:15
        volumeMounts:
        - name: pgdata
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: pgdata
    spec:
      storageClassName: fast-ssd
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 100Gi

Scaling a StatefulSet: Why Your Database Will Thank You for Ordered Pod Termination

Scaling a StatefulSet is not like scaling a Deployment. Deployment scaling is stupid-fast: it kills or spawns pods in parallel. StatefulSet scaling is surgical and ordered, and that matters when your app has a quorum.

When you scale down from 3 to 1 replica, Kubernetes terminates the highest ordinal pod first — postgres-cluster-2, then postgres-cluster-1. It waits for each pod to fully terminate (status.phase: Succeeded or Failed) before touching the next. No race conditions. No two databases trying to write to the same volume.

There's a hidden gotcha: kubectl scale statefulset postgres-cluster --replicas=0 doesn't delete the PVCs. The volumes hang around with their data intact. This is by design — you can scale back up and the pods reattach to the same disks. But if you manually delete PVCs while the pods are gone, you lose data permanently.

For applications that don't care about ordering (like a distributed cache), set spec.podManagementPolicy: Parallel — scaling goes from O(n) to O(1) time. But for databases, leave it as OrderedReady and let the controller handle the serial dance.

// io.thecodeforge — devops tutorial

# Scale down in action
$ kubectl scale statefulset postgres-cluster --replicas=2

# Kubernetes terminates postgres-cluster-2 first
# Watches pod status until it's gone
$ kubectl get pods -l app=postgres -w
NAME                  READY   STATUS        RESTARTS   AGE
postgres-cluster-0    1/1     Running       0          15m
postgres-cluster-1    1/1     Running       0          14m
postgres-cluster-2    1/1     Terminating   0          13m
postgres-cluster-2    0/1     Terminating   0          13m
postgres-cluster-2    0/1     Terminated    0          13m

OnDelete: Manual Pod Replacement When You Control the Timing

The OnDelete update strategy exists because rolling updates are dangerous for stateful workloads. Unlike RollingUpdate, which replaces pods in reverse ordinal order automatically, OnDelete requires you to delete each pod manually before it gets recreated with the new spec. Why use this? When upgrading a database cluster, you may need to drain connections, wait for replication lag to subside, or run pre-upgrade health checks on each node. OnDelete gives you precise per-pod control over when updates happen. Set updateStrategy.type: OnDelete in your StatefulSet spec. After updating the pod template, no pods change until you run kubectl delete pod <pod-name>. The controller then rebuilds that pod using the new template, preserving its identity and storage. This prevents the controller from deciding the pace of your production upgrade.

// io.thecodeforge — devops tutorial

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-cluster
spec:
  serviceName: postgres-hl
  replicas: 3
  updateStrategy:
    type: OnDelete
  template:
    spec:
      containers:
      - name: postgres
        image: postgres:16
        ports:
        - containerPort: 5432
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

Cleaning Up: Why StatefulSets Refuse to Die Cleanly

Deleting a StatefulSet does not automatically delete its PersistentVolumeClaims. By design, PVCs survive deletion to prevent accidental data loss. Run kubectl delete statefulset <name> and pods terminate in reverse order (pod-2, pod-1, pod-0), but the PVCs remain. If you recreate the StatefulSet with the same name and volumeClaimTemplates, it will reclaim those exact PVCs. To fully clean up, you must delete the PVCs manually after the StatefulSet is gone: kubectl delete pvc -l app=<your-app>. For cascading cleanup, set persistentVolumeClaimRetentionPolicy to delete in the StatefulSet spec. This policy defines what happens to PVCs when the StatefulSet is scaled down or deleted. With whenDeleted: Delete, PVCs are removed automatically. Without this, orphaned PVCs can rack up cloud storage costs indefinitely. Always audit orphaned resources after cleanup.

// io.thecodeforge — devops tutorial

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
spec:
  serviceName: redis-hl
  replicas: 3
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Delete
    whenScaled: Retain
  template:
    spec:
      containers:
      - name: redis
        image: redis:7
        ports:
        - containerPort: 6379
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 5Gi