Kubernetes StatefulSets — Kafka's 14-Hour Split-Brain

Stateless apps are easy. You spin up ten identical pods, kill any three, Kubernetes replaces them and nobody cares which replacement is which. But the real world is full of systems that refuse to be stateless: databases, message brokers, distributed caches, search engines. These systems have opinions about identity. Elasticsearch node 2 needs to rejoin the cluster as Elasticsearch node 2 — not as some random newcomer that triggers a full shard rebalance. Kafka broker 0 owns specific partitions and cannot pretend to be a fresh broker without corrupting the consumer offset records for every topic it leads. Ignoring this reality and running stateful workloads as Deployments is one of the most expensive mistakes teams make on Kubernetes, and it almost always surfaces at 2am during a production incident when an on-call engineer is staring at split-brain metrics they do not immediately understand.

StatefulSets exist precisely to give Kubernetes the vocabulary to reason about identity, ordering, and sticky storage. They provide three guarantees that Deployments fundamentally cannot: a stable, unique network identity that survives pod restarts and rescheduling; stable, persistent storage that follows the pod regardless of which node it lands on; and ordered, graceful deployment and scaling that respects cluster quorum requirements. These are not conveniences — they are load-bearing architectural properties that distributed consensus protocols depend on at the wire level.

By the end of this article you will understand how StatefulSets work under the hood: the controller loop, the role of the headless service in DNS, how PVC ownership is tracked via OwnerReferences, why pod ordinals matter for rolling updates, and the exact failure modes that bite teams in production. You will also have complete, runnable manifests with every significant field explained.

What is a Kubernetes StatefulSet?

A StatefulSet is a Kubernetes workload API object designed specifically for managing stateful applications. The name is deliberately chosen to contrast with the default Kubernetes mental model — stateless pods that are interchangeable, ephemeral, and replaceable. StatefulSets break that model intentionally and provide three guarantees that distributed stateful systems depend on at the protocol level.

First: stable, unique network identity. Each pod in a StatefulSet receives a deterministic name based on its ordinal index — web-0, web-1, web-2. This name is not a random hash suffix. It survives pod restarts, node failures, and rescheduling. Combined with a headless Service, this creates a stable DNS entry that other pods can use to discover and connect to a specific instance — which is critical for systems like ZooKeeper, etcd, and Kafka where cluster membership is a first-class concept baked into the protocol.

Second: stable, persistent storage. volumeClaimTemplates provision a dedicated PersistentVolumeClaim per pod using a naming convention tied to the pod's ordinal. When a pod is deleted and recreated, it reattaches to the same PVC. The storage follows the identity, not the node.

Third: ordered deployment and scaling. Pods are created sequentially — pod 0 must be Running and Ready before pod 1 is created. Pods are deleted in reverse order — pod N-1 before pod N-2 before pod 0. This ordering is not a soft preference — it is enforced by the StatefulSet controller at the API level.

Understanding which of these three guarantees your workload actually needs is more important than knowing the YAML syntax. A single-instance PostgreSQL database needs storage persistence but does not need the ordering guarantees. A ZooKeeper ensemble needs all three.

# The headless Service must be created BEFORE the StatefulSet.
# The StatefulSet controller validates the serviceName reference on creation.
# Without the headless Service, per-pod DNS records cannot be created.
apiVersion: v1
kind: Service
metadata:
  name: io-thecodeforge-headless
  labels:
    app: io-thecodeforge-db
spec:
  clusterIP: None  # This is what makes it headless — no virtual IP, direct pod DNS records
  selector:
    app: io-thecodeforge-db  # Must match the StatefulSet pod labels exactly
  ports:
    - port: 5432
      name: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: io-thecodeforge-db
spec:
  serviceName: io-thecodeforge-headless  # References the headless Service above
  replicas: 3
  selector:
    matchLabels:
      app: io-thecodeforge-db
  template:
    metadata:
      labels:
        app: io-thecodeforge-db
    spec:
      terminationGracePeriodSeconds: 120  # Give PostgreSQL time to flush WAL before SIGKILL
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: io-thecodeforge-db-secret
                  key: password
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "postgres"]
            initialDelaySeconds: 10
            periodSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  # volumeClaimTemplates creates one PVC per pod:
  # data-io-thecodeforge-db-0, data-io-thecodeforge-db-1, data-io-thecodeforge-db-2
  # These PVCs survive pod deletion and are reattached when pods are recreated.
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]  # One node at a time — appropriate for databases
        storageClassName: io-thecodeforge-ssd
        resources:
          requests:
            storage: 50Gi

Ordered Operations: Creation, Deletion, and Rolling Updates

StatefulSet operations are strictly ordered in ways that feel unusual if you are used to Deployment behaviour. Understanding this ordering is essential for both operating StatefulSets correctly and for debugging when something gets stuck.

Pod creation is strictly sequential: the controller creates pod 0, then waits until it is Running and Ready before creating pod 1. Pod 1 must be Running and Ready before pod 2 is created. This is not just about startup — it reflects the dependency structure of many distributed systems where node 0 bootstraps the cluster and node 1 joins as a follower. If you scale a StatefulSet from 3 to 5 replicas, pods 3 and 4 are created sequentially in that order.

Pod deletion is strictly reverse-sequential: the controller deletes pod N-1, waits for it to fully terminate, then deletes pod N-2, and so on to pod 0. This preserves quorum during scale-down — a ZooKeeper ensemble being scaled from 5 to 3 nodes loses its two highest-ordinal nodes first, maintaining the 3-node quorum throughout the process rather than potentially losing the primary.

Rolling updates follow a special reverse-ordinal pattern that is different from what most engineers expect. The controller updates pod N-1 first, waits for it to become Ready, then proceeds to pod N-2, down to pod 0 last. Pod 0, which often holds the most critical role in distributed systems (initial voter in ZooKeeper, partition leader for critical topics in Kafka), is updated last — preserving cluster stability for as long as possible during the rollout.

The partition field in the rolling update strategy is one of the most useful and most forgotten StatefulSet features. Setting partition: 3 in a 5-pod StatefulSet means only pods 3 and 4 update when you apply a new template. Pods 0, 1, and 2 remain on the old version. This is a native canary mechanism built into the StatefulSet API — you validate pods 3 and 4 under production traffic, then lower partition to 0 to complete the rollout. The failure mode to remember: if you set partition for a canary and forget to reset it to 0, your cluster runs in a permanently split-version state indefinitely.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: io-thecodeforge-db
spec:
  serviceName: io-thecodeforge-headless
  replicas: 5
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      # Only pods with ordinal >= 3 will be updated when the template changes.
      # Pods 0, 1, and 2 remain on the current version.
      # This is the canary pattern: validate pods 3 and 4 first.
      # To complete the rollout: kubectl patch statefulset io-thecodeforge-db
      #   --type=merge -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'
      partition: 3
  selector:
    matchLabels:
      app: io-thecodeforge-db
  template:
    metadata:
      labels:
        app: io-thecodeforge-db
    spec:
      # 90 seconds gives PostgreSQL enough time to:
      # 1. Finish in-progress transactions
      # 2. Flush dirty buffers to disk
      # 3. Write a clean checkpoint to WAL
      # 4. Cleanly release file locks
      # Without sufficient grace period, the next startup triggers crash recovery.
      terminationGracePeriodSeconds: 90
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "postgres"]
            initialDelaySeconds: 10
            periodSeconds: 5
            # failureThreshold * periodSeconds = how long before the rollout is blocked
            # 3 * 5 = 15 seconds of failed readiness before the controller stops advancing
            failureThreshold: 3
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: io-thecodeforge-ssd
        resources:
          requests:
            storage: 50Gi

# Monitor rollout progress:
# kubectl rollout status statefulset/io-thecodeforge-db
# kubectl get statefulset io-thecodeforge-db -o jsonpath=\
#   '{.status.currentRevision} {.status.updateRevision}'

PVC Lifecycle: Storage That Follows Identity

StatefulSets use volumeClaimTemplates to provision one PersistentVolumeClaim per pod. The naming convention is deterministic: template-name-statefulset-name-ordinal. For a StatefulSet named io-thecodeforge-db with a template named data, the PVCs are data-io-thecodeforge-db-0, data-io-thecodeforge-db-1, and data-io-thecodeforge-db-2. This naming is not configurable — it is generated by the StatefulSet controller.

The storage lifecycle has two properties that every engineer running StatefulSets in production must understand deeply.

First, PVCs survive pod deletion. When a pod is deleted — whether by a rolling update, a manual kubectl delete pod, a node failure, or a scale-down — its PVC is not deleted. When the pod is recreated with the same ordinal, it reattaches to the same PVC. This is the 'sticky storage' guarantee. It is what makes your PostgreSQL data directory survive node failures without data loss.

Second, PVCs are NOT deleted when the StatefulSet is deleted. This is a deliberate safety mechanism — accidentally deleting a StatefulSet should not destroy production databases. But it means that scaling down a StatefulSet from 5 to 3 replicas leaves two orphaned PVCs (data-name-3 and data-name-4) that consume cloud storage indefinitely. At $0.08-0.15 per GB per month on most cloud providers, a 1TB database with ten orphaned scale-down PVCs accumulates $800-1500 per month in silent storage waste. This is one of the most common cost anomalies in Kubernetes clusters and one of the least visible.

Kubernetes 1.27 introduced the persistentVolumeClaimRetentionPolicy field on StatefulSets, which allows you to configure automatic PVC deletion on scale-down (whenScaled) or StatefulSet deletion (whenDeleted). For most production stateful workloads, setting whenScaled to Delete is appropriate — the PVC for a scaled-down pod can reasonably be considered ephemeral. Setting whenDeleted to Delete is more dangerous and should only be used when you have confirmed out-of-cluster backups.

# Kubernetes 1.27+ introduces persistentVolumeClaimRetentionPolicy
# This controls what happens to PVCs when pods are deleted or the StatefulSet scales down.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: io-thecodeforge-db
spec:
  serviceName: io-thecodeforge-headless
  replicas: 3
  # Available from Kubernetes 1.27+ (stable in 1.29+)
  persistentVolumeClaimRetentionPolicy:
    # whenScaled: what happens to PVCs when replicas is reduced
    #   Delete  - PVC is deleted when its pod is scaled away (use with confirmed backups)
    #   Retain  - PVC survives scale-down (default, safe but causes orphaned volume drift)
    whenScaled: Retain
    # whenDeleted: what happens to PVCs when the StatefulSet is deleted
    #   Delete  - PVCs are deleted with the StatefulSet (DANGEROUS — only with backups)
    #   Retain  - PVCs survive StatefulSet deletion (default, safe)
    whenDeleted: Retain
  selector:
    matchLabels:
      app: io-thecodeforge-db
  template:
    metadata:
      labels:
        app: io-thecodeforge-db
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: io-thecodeforge-ssd
        resources:
          requests:
            storage: 50Gi

---
# Manual PVC lifecycle management for clusters on Kubernetes < 1.27:

# After scaling down from 5 to 3 replicas, list orphaned PVCs:
# kubectl get pvc -l app=io-thecodeforge-db
# NAME                           STATUS  CAPACITY
# data-io-thecodeforge-db-0      Bound   50Gi   <- still in use
# data-io-thecodeforge-db-1      Bound   50Gi   <- still in use
# data-io-thecodeforge-db-2      Bound   50Gi   <- still in use
# data-io-thecodeforge-db-3      Bound   50Gi   <- ORPHANED (pod deleted)
# data-io-thecodeforge-db-4      Bound   50Gi   <- ORPHANED (pod deleted)

# Clean up orphaned PVCs after confirming data is not needed:
# kubectl delete pvc data-io-thecodeforge-db-3 data-io-thecodeforge-db-4

# WARNING: PVC deletion is IRREVERSIBLE
# The underlying PersistentVolume and its data are gone permanently.
# Always confirm you have a recent backup before deleting any database PVC.

Headless Services and DNS: How Pods Find Each Other

A headless Service is the DNS backbone of a StatefulSet. Without it, StatefulSet pods have stable names but no way for other pods to resolve those names to IP addresses. With it, every pod in the StatefulSet gets an individual DNS A record that points directly to that pod's IP — bypassing the load-balancing layer that regular Services add.

The distinction between a regular Service and a headless Service is important to internalise. A regular Service (clusterIP: something) creates one DNS name that resolves to a virtual IP, and kube-proxy load-balances traffic from that virtual IP to any matching pod. You cannot target a specific pod by DNS with a regular Service. A headless Service (clusterIP: None) creates no virtual IP and no load balancing. Instead, CoreDNS creates individual A records for each pod, one per pod, each pointing to that pod's actual IP address. This is how pod-to-pod targeting works.

The DNS naming convention for StatefulSet pods is: pod-name.service-name.namespace.svc.cluster.local. For a StatefulSet named db with a headless Service named db-headless in namespace prod, pod 0's DNS entry is db-0.db-headless.prod.svc.cluster.local. This entry is stable — it always points to the pod with that identity, regardless of which node it runs on. When the pod is rescheduled to a different node with a different IP, CoreDNS updates the A record to reflect the new IP. The DNS name remains constant; only the IP it resolves to changes.

The headless Service also creates a SRV record for each pod: _port-name._protocol.service-name.namespace.svc.cluster.local. SRV records carry both the hostname and port, which is how distributed systems like Kafka and ZooKeeper bootstrap cluster membership at startup — they query the SRV record to discover all current pod hostnames without needing to know ordinal count in advance.

One practical detail: CoreDNS caches DNS responses with a TTL. For headless Services the default TTL is 5 seconds. If a pod is rescheduled quickly, there is a brief window where other pods may try to connect to the old IP before the cache expires. Design your application's connection retry logic to tolerate this — most database connection pools handle it correctly if configured with appropriate connection timeouts.

# The headless Service — created before the StatefulSet, referenced by serviceName field.
apiVersion: v1
kind: Service
metadata:
  name: io-thecodeforge-headless
  namespace: prod
  labels:
    app: io-thecodeforge-db
spec:
  clusterIP: None  # No virtual IP — each pod gets its own DNS A record
  publishNotReadyAddresses: false  # Only Ready pods get DNS records (default: false)
  # Set to true for stateful systems that need to discover all members
  # including those still initializing — useful for ZooKeeper ensemble bootstrap
  selector:
    app: io-thecodeforge-db  # Must match StatefulSet pod labels EXACTLY
  ports:
    - port: 5432
      targetPort: 5432
      name: postgres

# --- DNS Records Created by CoreDNS ---
#
# Per-pod A records (stable, direct — the primary use case):
#   io-thecodeforge-db-0.io-thecodeforge-headless.prod.svc.cluster.local -> 10.244.1.5
#   io-thecodeforge-db-1.io-thecodeforge-headless.prod.svc.cluster.local -> 10.244.2.8
#   io-thecodeforge-db-2.io-thecodeforge-headless.prod.svc.cluster.local -> 10.244.3.12
#
# SRV records (used for automatic cluster member discovery):
#   _postgres._tcp.io-thecodeforge-headless.prod.svc.cluster.local
#   -> SRV: 0 50 5432 io-thecodeforge-db-0.io-thecodeforge-headless.prod.svc.cluster.local
#   -> SRV: 0 50 5432 io-thecodeforge-db-1.io-thecodeforge-headless.prod.svc.cluster.local
#   -> SRV: 0 50 5432 io-thecodeforge-db-2.io-thecodeforge-headless.prod.svc.cluster.local
#
# Service-level A record (returns all pod IPs — round-robin, not load-balanced):
#   io-thecodeforge-headless.prod.svc.cluster.local -> 10.244.1.5, 10.244.2.8, 10.244.3.12

# --- Verify DNS resolution from inside the cluster ---
# kubectl exec io-thecodeforge-db-1 -- \n#   nslookup io-thecodeforge-db-0.io-thecodeforge-headless.prod.svc.cluster.local
# Expected output: Address: 10.244.1.5

PodDisruptionBudget: Protecting Quorum During Voluntary Disruptions

PodDisruptionBudgets (PDBs) are among the most important and most skipped Kubernetes objects for StatefulSets in production. A PDB is a policy object that limits how many pods in a set can be simultaneously unavailable during voluntary disruptions — node drains, cluster upgrades, autoscaler scale-downs, and manual evictions.

For a quorum-based system, the requirement is concrete: a 3-node ZooKeeper cluster needs at least 2 nodes to maintain quorum. A 5-node etcd cluster needs at least 3 nodes. A PDB with minAvailable: 2 tells Kubernetes — during any voluntary disruption, you must ensure at least 2 of my pods are running and Ready before proceeding with the eviction. If a node drain would cause a second pod to become unavailable while the first is still evicting, the drain blocks until the first pod is rescheduled and Ready elsewhere.

This blocking behaviour is the point. Without a PDB, kubectl drain proceeds freely, evicting pods without regard for quorum. A node drain during a cluster upgrade with three StatefulSet pods all on the same node (a common situation if pod anti-affinity is not configured) will evict all three simultaneously, destroying quorum completely.

The combination of PDB plus pod anti-affinity is the production-grade pattern. The PDB protects against voluntary disruptions. Pod anti-affinity with requiredDuringSchedulingIgnoredDuringExecution and topologyKey: kubernetes.io/hostname protects against involuntary disruptions by ensuring no two pods land on the same node. Together, they mean: node drains cannot break quorum, and a single node failure cannot take out more than one pod.

One operational detail: kubectl drain respects PDBs. kubectl delete pod does not. If your operational runbook uses kubectl delete pod to perform maintenance, PDBs provide no protection. Use kubectl drain or kubernetes eviction API calls for any maintenance that should respect disruption budgets.

# PodDisruptionBudget — protects quorum during voluntary disruptions
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: io-thecodeforge-db-pdb
  namespace: prod
spec:
  # For a 3-node cluster: minAvailable: 2 (majority = quorum)
  # For a 5-node cluster: minAvailable: 3 (majority = quorum)
  # For a 7-node cluster: minAvailable: 4 (majority = quorum)
  # Rule: minAvailable = floor(replicas / 2) + 1
  minAvailable: 2
  selector:
    matchLabels:
      app: io-thecodeforge-db

# Alternative using maxUnavailable (equivalent for a 3-pod set):
# spec:
#   maxUnavailable: 1
#   selector:
#     matchLabels:
#       app: io-thecodeforge-db

---
# Pod anti-affinity — spreads pods across nodes to protect against involuntary failures
# PDB protects against voluntary disruptions (drain, upgrade)
# Anti-affinity protects against involuntary disruptions (node crash)
# You need BOTH for production-grade StatefulSet resilience.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: io-thecodeforge-db
  namespace: prod
spec:
  serviceName: io-thecodeforge-headless
  replicas: 3
  selector:
    matchLabels:
      app: io-thecodeforge-db
  template:
    metadata:
      labels:
        app: io-thecodeforge-db
    spec:
      affinity:
        podAntiAffinity:
          # required = hard constraint: pods WILL NOT be scheduled on the same node
          # preferred = soft preference: Kubernetes tries but can violate if necessary
          # For quorum-sensitive systems, use required.
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - io-thecodeforge-db
              # kubernetes.io/hostname = spread across nodes
              # topology.kubernetes.io/zone = spread across AZs (stronger guarantee)
              topologyKey: topology.kubernetes.io/zone
      containers:
        - name: postgres
          image: postgres:16-alpine
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: io-thecodeforge-ssd
        resources:
          requests:
            storage: 50Gi