Kubernetes StatefulSets: PVC Orphan Caused 2TB Leak

Stateless apps are easy. You spin up ten identical pods, kill any three, Kubernetes replaces them — nobody cares. But the real world is full of systems that refuse to be stateless: databases, message brokers, distributed caches, search engines. These systems have opinions. Elasticsearch node 2 needs to rejoin the cluster as Elasticsearch node 2, not as some random newcomer. Kafka broker 0 owns specific partitions and cannot pretend to be a fresh broker without corrupting data.

StatefulSets exist precisely to give Kubernetes the vocabulary to reason about identity, ordering, and sticky storage. They provide three guarantees that Deployments fundamentally cannot: a stable, unique network identity that survives pod restarts; stable, persistent storage that follows the pod around regardless of which node it lands on; and ordered, graceful deployment and scaling.

This is not a getting-started guide. It covers the controller loop internals, PVC ownership tracking, the role of the Headless Service, why pod ordinals matter for rolling updates, and the exact failure modes that bite teams in production.

How Kubernetes Services and Ingress Actually Route Traffic

A Kubernetes Service is a stable network endpoint that decouples client traffic from a dynamic set of Pods. It provides a virtual IP (ClusterIP) or external load balancer that forwards requests to healthy Pods based on label selectors. Ingress, on the other hand, is an API object that manages external HTTP/S access to Services, offering host-based and path-based routing, TLS termination, and virtual hosting — all within a single external endpoint. Together, they form the traffic routing layer of a Kubernetes cluster.

Services operate at Layer 4 (TCP/UDP) by default, using iptables or IPVS rules to distribute traffic across Pods. Ingress controllers (e.g., NGINX, Traefik, HAProxy) implement the Ingress resource at Layer 7, parsing HTTP headers and paths to route to the correct Service. This separation means you can scale Pods independently without changing client configurations — the Service IP remains constant, and Ingress rules update dynamically as Services are added or removed.

Use Services when you need internal cluster communication or simple TCP/UDP load balancing. Add Ingress when you need HTTP-specific features like path-based routing, SSL termination, or multiple domains on a single IP. In production, this pattern is essential for zero-downtime deployments: Services abstract Pod churn, and Ingress enables canary releases by routing a percentage of traffic to a different Service.

StatefulSet vs Deployment: When to Use Each

The most common decision Kubernetes operators face is choosing between a Deployment and a StatefulSet. Both manage replica Pods, but they differ fundamentally in how they treat identity, storage, and order.

A Deployment assumes all Pods are interchangeable. Each Pod gets a random name (e.g., myapp-68dcf7d8b4-abc123), can be replaced with any other Pod, and uses ephemeral or shared storage. Deployments scale quickly in parallel and are perfect for stateless services: web servers, REST APIs, worker queues.

A StatefulSet assumes each Pod is unique. Each Pod gets a stable ordinal name (pod-0, pod-1), a persistent DNS entry via a Headless Service, and its own PersistentVolumeClaim that follows it across reschedules. StatefulSets scale one Pod at a time (ordered) and are necessary for stateful workloads: databases (PostgreSQL, Cassandra), message brokers (Kafka, RabbitMQ), distributed consensus systems (etcd, ZooKeeper).

If you are unsure, start with a Deployment. Stateless applications are simpler to scale, debug, and upgrade. Only move to a StatefulSet when you encounter a concrete requirement for stable identity or per-Pod storage. Overusing StatefulSets adds unnecessary complexity and cost.

When neither fits, consider a DaemonSet for running exactly one Pod per node (e.g., log collectors, node monitoring agents) or a Job/CronJob for batch workloads.

# Deployment for stateless API
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: myapp:1.0
          ports:
            - containerPort: 8080
---
# StatefulSet for Kafka broker
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka-hs
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
        - name: kafka
          image: confluentinc/cp-kafka:7.5.0
          ports:
            - containerPort: 9092
          volumeMounts:
            - name: data
              mountPath: /var/lib/kafka/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Gi

Kubernetes Service Types: ClusterIP, NodePort, LoadBalancer, and Headless

Services abstract Pod-to-Pod communication and external access. Each Service type serves a different purpose and carries different trade-offs. Understanding them is essential for exposing StatefulSet Pods correctly.

ClusterIP (default): Exposes the Service on an internal cluster IP. Only reachable from within the cluster. Use for internal microservice communication. ClusterIP is the most efficient because it does not require external load balancers.

NodePort: Exposes the Service on each Node's IP at a static port (30000-32767). Reachable from outside the cluster by hitting <NodeIP>:<NodePort>. Use for development, debugging, or when you need direct node access. Not recommended for production due to security and port collision issues.

LoadBalancer: Exposes the Service externally via a cloud provider's load balancer (e.g., AWS ELB, GCP HTTP(S) LB). Automatically creates a NodePort and ClusterIP behind the scenes. Use for exposing a single Service to the internet. Each Service gets its own load balancer, which incurs hourly cost.

Headless (clusterIP: None): Does not allocate a cluster IP. DNS returns the IPs of all healthy Pods directly. Used primarily with StatefulSets for per-Pod DNS records (pod-0.service.ns.svc.cluster.local). Clients decide which Pod to contact. Also used for custom service discovery.

When exposing a StatefulSet externally, you typically use a LoadBalancer or Ingress for the entire cluster (all Pods), not per-Pod. For inter-Pod communication within the StatefulSet, you always use a Headless Service.

# ClusterIP: internal only
apiVersion: v1
kind: Service
metadata:
  name: my-internal
spec:
  selector:
    app: myapp
  ports:
    - port: 80
      targetPort: 8080
---
# NodePort: external via node IP
apiVersion: v1
kind: Service
metadata:
  name: my-nodeport
spec:
  type: NodePort
  selector:
    app: myapp
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30080
---
# LoadBalancer: external via cloud LB
apiVersion: v1
kind: Service
metadata:
  name: my-lb
spec:
  type: LoadBalancer
  selector:
    app: myapp
  ports:
    - port: 80
      targetPort: 8080
---
# Headless: no cluster IP, used with StatefulSets
apiVersion: v1
kind: Service
metadata:
  name: my-headless
spec:
  clusterIP: None
  selector:
    app: myapp
  ports:
    - port: 80
      targetPort: 8080

Ingress vs LoadBalancer: Choosing the Right External Exposure Mechanism

When you need to expose a StatefulSet (or any service) to the internet, you have two primary options: a LoadBalancer Service or an Ingress resource. The choice depends on protocol, routing requirements, and cost.

LoadBalancer Service: Creates a cloud load balancer (e.g., AWS ELB, GCP TCP LB) that forwards traffic directly to your Service's Pods. Operates at Layer 4 (TCP/UDP). Each LoadBalancer Service gets its own static IP or DNS name. Simple to set up, but each one is a separate billable resource. Best for non-HTTP protocols (gRPC, WebSocket, database connections) or when you need a single service exposed with minimal configuration.

Ingress: A cluster-level resource that provides HTTP(S) routing rules. Requires an Ingress controller (e.g., NGINX Ingress, Istio Gateway, AWS Load Balancer Controller). The controller typically runs as a DaemonSet or Deployment and is itself exposed via a LoadBalancer Service. Ingress operates at Layer 7, allowing path-based routing (e.g., /api -> service-a, /web -> service-b), host-based routing (api.example.com -> service-a), SSL termination, and rate limiting. Multiple Ingress rules can share the same underlying LoadBalancer, saving money.

Decision Matrix: | Criteria | LoadBalancer Service | Ingress | |----------|---------------------|---------| | Protocol | TCP, UDP, HTTP | HTTP, HTTPS, gRPC (with controller) | | Routing | No (single target) | Path, host, headers | | SSL termination | Manual (annotation) | Built-in (cert-manager) | | Cost per service | One LB per Service | One LB for many Ingresses | | Setup complexity | Low | Medium (controller required) | | Use case | Database, non-HTTP, simple apps | HTTP APIs, web apps, microservices |

For HTTP workloads with multiple services, use Ingress. For non-HTTP workloads or when you need absolute simplicity, use LoadBalancer. You can also combine both: an Ingress controller exposed via a LoadBalancer Service.

# LoadBalancer Service for a database (non-HTTP)
apiVersion: v1
kind: Service
metadata:
  name: postgres-lb
spec:
  type: LoadBalancer
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432
---
# Ingress for HTTP API (multiple services behind one LB)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /users
            pathType: Prefix
            backend:
              service:
                name: user-service
                port:
                  number: 8080
          - path: /orders
            pathType: Prefix
            backend:
              service:
                name: order-service
                port:
                  number: 8080

Stable Identity: Network Names and Pod Ordinals

The defining feature of a StatefulSet is stable identity. Each Pod receives a unique, predictable name based on the StatefulSet name and an ordinal index: <statefulset-name>-0, <statefulset-name>-1, <statefulset-name>-2. This identity persists across restarts, reschedules, and even node failures. If pod-2 is rescheduled to a different node, it is still pod-2.

This identity extends to DNS. When a StatefulSet specifies a Headless Service via spec.serviceName, Kubernetes creates DNS A records for each Pod: pod-0.service.namespace.svc.cluster.local. These DNS names resolve directly to the Pod's IP address. When the Pod restarts with a new IP, the DNS record is updated automatically.

This is fundamentally different from Deployments, where Pods are interchangeable and have random names. Stateful systems rely on this identity for peer discovery, leader election, and data partitioning. A Kafka broker must rejoin the cluster with the same identity to resume ownership of its partitions.

# StatefulSet with stable identity and Headless Service.
# Each Pod is reachable as: postgres-0.postgres.production.svc.cluster.local
# Package: io.thecodeforge.k8s

# Headless Service: DNS returns Pod IPs directly.
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: production
spec:
  clusterIP: None              # Headless: no virtual IP
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432
---
# StatefulSet: stable identity, ordered operations, sticky storage.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: production
spec:
  serviceName: postgres        # Must match the Headless Service name
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:15
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-credentials
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec:
              command:
                - pg_isready
                - -U
                - postgres
            initialDelaySeconds: 5
            periodSeconds: 5
  # volumeClaimTemplates: creates a PVC for each Pod.
  # PVC name format: <template-name>-<statefulset-name>-<ordinal>
  # e.g., data-postgres-0, data-postgres-1, data-postgres-2
  # These PVCs persist even after the StatefulSet is deleted.
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 100Gi

Ordered Operations: Creation, Deletion, and Rolling Updates

StatefulSets enforce strict ordering on all lifecycle operations. Pods are created sequentially from ordinal 0 to N-1. Pod N is not created until Pod N-1 is Running and Ready. Pods are deleted in reverse order: N-1 first, then N-2, down to 0. Rolling updates follow the same ordinal order.

This ordering is critical for systems that need quorum during bootstrap. A 3-node etcd cluster needs at least 2 nodes to form quorum. If all 3 Pods start simultaneously, none can form quorum because they all try to discover peers that do not exist yet. Ordered creation ensures pod-0 starts first, pod-1 joins pod-0, and pod-2 joins the existing 2-node cluster.

The podManagementPolicy field controls this behavior. The default is OrderedReady: Pods are created and deleted one at a time in ordinal order. The alternative is Parallel: Pods are created and deleted simultaneously, like a Deployment. Parallel is faster but bootstrap.

# StatefulSet with OrderedReady (default) and Parallel comparison.
# Package: io.thecodeforge.k8s

# OrderedReady: Pods created one at a time. Slow but safe for quorum-based systems.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: etcd-cluster
  namespace: production
spec:
  serviceName: etcd
  replicas: 3
  podManagementPolicy: OrderedReady  breaks systems that need ordered # Default. Sequential creation/deletion.
  selector:
    matchLabels:
      app: etcd
  template:
    metadata:
      labels:
        app: etcd
    spec:
      containers:
        - name: etcd
          image: quay.io/coreos/etcd:v3.5.12
          ports:
            - containerPort: 2379
              name: client
            - containerPort: 2380
              name: peer
          env:
            - name: ETCD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: ETCD_INITIAL_CLUSTER
              value: "etcd-0=http://etcd-0.etcd:2380,etcd-1=http://etcd-1.etcd:2380,etcd-2=http://etcd-2.etcd:2380"
            - name: ETCD_INITIAL_CLUSTER_STATE
              value: "new"
            - name: ETCD_INITIAL_CLUSTER_TOKEN
              value: "etcd-cluster-token"
            - name: ETCD_DATA_DIR
              value: "/var/lib/etcd/data"
          volumeMounts:
            - name: data
              mountPath: /var/lib/etcd
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi
---
# Parallel: All Pods created simultaneously. Fast but unsafe for quorum bootstrap.
# Use only for systems that do not require ordered startup.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cache
  namespace: production
spec:
  serviceName: redis
  replicas: 3
  podManagementPolicy: Parallel    # All Pods created at once.
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

PersistentVolumeClaim Lifecycle: Ownership, Orphans, and Reclaim

The volumeClaimTemplates field in a StatefulSet is a template for creating PVCs. When a StatefulSet Pod is created, Kubernetes creates a PVC from the template with a deterministic name: <template-name>-<statefulset-name>-<ordinal>. For example, a StatefulSet named postgres with a volumeClaimTemplate named data creates PVCs: data-postgres-0, data-postgres-1, data-postgres-2.

These PVCs are owned by the StatefulSet but are NOT deleted when the StatefulSet is deleted. This is by design — the data must persist so it can be re-attached if the StatefulSet is re-created. However, this creates a common production pitfall: orphaned PVCs that consume storage indefinitely.

The reclaim policy on the underlying StorageClass determines what happens to the PersistentVolume when the PVC is finally deleted. Retain keeps the PV and its data. Delete removes the PV and its data. The default varies by cloud provider.

# PVC lifecycle management for StatefulSets.
# Package: io.thecodeforge.k8s

# StorageClass with Delete reclaim policy.
# When PVC is deleted, the underlying PV and data are also deleted.
# Use for ephemeral or reproducible data.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd-deletable
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
reclaimPolicy: Delete          # PV is deleted when PVC is deleted
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
# StorageClass with Retain reclaim policy.
# When PVC is deleted, the PV is kept (but unbound).
# Use for critical data that must survive PVC deletion.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd-retain
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
reclaimPolicy: Retain          # PV is kept when PVC is deleted
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
# StatefulSet using the deletable storage class.
# PVCs are auto-deleted when the StatefulSet is deleted IF you also delete the PVCs.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: production
spec:
  serviceName: kafka
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
        - name: kafka
          image: confluentinc/cp-kafka:7.5.0
          ports:
            - containerPort: 9092
          env:
            - name: KAFKA_BROKER_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['apps.kubernetes.io/pod-index']
            - name: KAFKA_LOG_DIRS
              value: /var/lib/kafka/data
          volumeMounts:
            - name: data
              mountPath: /var/lib/kafka/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd-deletable
        resources:
          requests:
            storage: 500Gi

Update Strategies: RollingUpdate vs OnDelete

StatefulSets support two update strategies: RollingUpdate (default) and OnDelete. The choice determines how Pod template changes (image update, env var change) are propagated to existing Pods.

RollingUpdate updates Pods one at a time in ordinal order, waiting for each Pod to be Ready before proceeding to the next. This is the safe default but slow. The maxUnavailable field (available in Kubernetes 1.24+) controls how many Pods can be unavailable during the update.

OnDelete does not automatically update Pods. When the Pod template is changed, existing Pods continue running the old spec. The update is applied only when a Pod is manually deleted. Kubernetes recreates the Pod with the new spec. This gives full control over update timing but requires manual intervention.

# StatefulSet update strategies.
# Package: io.thecodeforge.k8s

# RollingUpdate (default): Automatic, ordered updates.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zookeeper
  namespace: production
spec:
  serviceName: zookeeper
  replicas: 3
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1        # Allow 1 Pod to be unavailable during update
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
        - name: zookeeper
          image: zookeeper:3.8
          ports:
            - containerPort: 2181
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - "echo ruok | nc localhost 2181 | grep imok"
            initialDelaySeconds: 10
            periodSeconds: 5
          volumeMounts:
            - name: data
              mountPath: /data
            - name: datalog
              mountPath: /datalog
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi
    - metadata:
        name: datalog
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi
---
# OnDelete: Manual update control. No automatic Pod recreation on template change.
apiVersion: apps/v1
kind: systems where each restart has significant operational impact (rebalancing, reindexing, replication catch-up).

PodDisruptionBudgets and StatefulSet Availability

PodDisruptionBudgets (PDBs) are critical for StatefulSets. They prevent the voluntary disruption controller from evicting too many Pods simultaneously during node drains, cluster upgrades, or preemptions. The controller does not intentionally evict more Pods than the budget allows.

For a 3-node etcd cluster, set minAvailable: 2. This ensures that a node drain cannot break quorum. If the drain would evict a third Pod, it blocks until one of the evicted Pods is rescheduled and Ready.

PDBs only block voluntary disruption (drain, upgrade, preemption). They do NOT protect against involuntary disruption (node crash, OOMKill, kernel panic). This distinction is critical: PDBs are a guardrail for planned maintenance, not a safety net for unplanned failures.

# PodDisruptionBudget for a 3-node etcd cluster.
# Ensures at least 2 Pods are always available (quorum).
# Package: io.thecodeforge.k8s
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: etcd-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: etcd
---
# PodDisruptionBudget using maxUnavailable for a 5-node Kafka cluster.
# Allows at most 1 Pod to be down at any time.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
  namespace: production
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: kafka

Ingress Is Dead. Long Live Gateway API.

The Ingress API is frozen. No new features. No bug fixes that aren't security critical. Kubernetes v1.19 marked it stable, and then the project stopped. Gateway API is the replacement. If you are building a new cluster, do not design around Ingress. Design around Gateway, even if you wrap it with an Ingress controller for compatibility. Why? Because Gateway separates the concerns Ingress smushed together: infrastructure provider, cluster operator, and application developer. Ingress forces a single resource to define routing, TLS, and backend selection. Gateway lets each team own their piece without stepping on each other. The production incident you will face: an app team changes an Ingress rule and accidentally takes down another team's routing because the Ingress object is shared. Gateway prevents that. Migrate now, not when your Ingress breaks and you have no upgrade path.

// io.thecodeforge
# Gateway API v1 example — not Ingress
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: prod-gateway
spec:
  gatewayClassName: nginx
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    hostname: "*.acme.io"
    tls:
      mode: Terminate
      certificateRefs:
        - name: wildcard-cert
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payments-api
spec:
  parentRefs:
  - name: prod-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /payments
    backendRefs:
    - name: payments-v2
      port: 8080

Path Types: The Silent Routing Breakage

Ingress supports three path types: Prefix, Exact, and ImplementationSpecific. The one that bites everyone is Prefix. It does not behave like a regex or glob. Prefix matching means it matches if the request path starts with the given path. A rule for /api will match /api/v1, /api/v2, and /api/v1/orders. But it also matches /api-keys because /api is a prefix of /api-keys. Your regex-loving brain says 'that's wrong'. Your nginx controller says 'too bad, it's kubernetes-native prefix matching'. Always append a trailing slash to your paths unless you explicitly want prefix behavior. Use Exact for literal endpoints. ImplementationSpecific delegates to your controller — meaning your rules are not portable. If you migrate from nginx to Contour, your ImplementationSpecific rules will break silently. Stick to Prefix and Exact for portability. Test with curl before deploying to production.

// io.thecodeforge
# Correct path matching — avoid the /api-keys trap
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-gateway
spec:
  ingressClassName: nginx
  rules:
  - host: api.acme.io
    http:
      paths:
      - path: /api/v1
        pathType: Prefix
        backend:
          service:
            name: v1-backend
            port:
              number: 80
      - path: /docs
        pathType: Exact
        backend:
          service:
            name: docs-static
            port:
              number: 80