Kubernetes Pods and Deployments - Missing Startup Probe 502

Every production Kubernetes workload ultimately runs as Pods managed by Deployments. The Pod is the atomic scheduling unit — a group of containers sharing a network namespace and volumes. The Deployment is the declarative controller that ensures the right number of Pods exist, handles rolling updates, and rolls back on failure.

Misconfiguring either object causes production incidents: Pods without resource limits cause noisy-neighbor OOMKills, Deployments without proper probes cause 502 errors during rollouts, and direct Pod creation bypasses self-healing entirely. Understanding the reconciliation loop — how the Deployment controller continuously drives current state toward desired state — is the foundation for debugging every higher-level Kubernetes object.

Pod Basics: The Atomic Unit

In the world of Kubernetes, the Pod is the atomic unit of scheduling. While you might be used to thinking in terms of 'containers,' Kubernetes thinks in 'Pods.' A Pod can host a single container, or a tightly coupled group of containers (like an app container and a 'sidecar' logging agent) that need to share the same local network (localhost) and storage volumes.

Crucially, Pods are ephemeral. They are born, they live, and they die. They are never 'repaired'; they are replaced.

# Package: io.thecodeforge.k8s.manifests
# pod.yaml — used primarily for local debugging, rarely in production
apiVersion: v1
kind: Pod
metadata:
  name: forge-api-pod
  labels:
    app: forge-api
    env: production
spec:
  containers:
    - name: api-container
      image: io.thecodeforge/api:1.2.0
      ports:
        - containerPort: 8080
      resources:
        requests:
          memory: "128Mi"
          cpu: "100m"
        limits:
          memory: "256Mi"
          cpu: "500m"
      env:
        - name: APP_COLOR
          value: "blue"
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: cloud-secrets
              key: db-password

Deployments: Orchestrating the Desired State

A Deployment is a high-level object that manages a ReplicaSet, which in turn manages Pods. Its job is to ensure the 'Desired State' matches the 'Current State.' If you tell a Deployment you want 3 replicas, and a node crashes taking one Pod with it, the Deployment controller notices the discrepancy and immediately schedules a new Pod on a healthy node.

Deployments are also the primary vehicle for Rolling Updates. By manipulating the maxSurge and maxUnavailable parameters, you can swap out version 1.0 for 2.0 without dropping a single user request.

# deployment.yaml — The production standard
apiVersion: apps/v1
kind: Deployment
metadata:
  name: forge-api-deployment
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: forge-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: forge-api
    spec:
      containers:
        - name: api
          image: io.thecodeforge/api:2.0.1
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

Probe Comparison: Liveness vs Readiness vs Startup

Kubernetes provides three distinct probe types — liveness, readiness, and startup — each serving a different purpose in the Pod lifecycle. Misunderstanding their roles is the root cause of many production incidents, including the 502 error scenario described earlier.

Liveness Probe: Determines if the container is alive. If it fails, the kubelet kills the container and restarts it (according to the Pod's restartPolicy). Use it to recover from deadlocks or infinite loops. Never check external dependencies in a liveness probe — if the database is slow, the probe fails, the container restarts, and the restart increases load on the database, causing cascading failure.

Readiness Probe: Determines if the container is ready to serve traffic. If it fails, the Pod's IP is removed from the Service endpoints — traffic stops, but the container is NOT restarted. Use it for applications that need to load cache, connect to databases, or run startup migrations before handling requests. The readiness probe controls traffic flow, not container lifecycle.

Startup Probe: Gates both liveness and readiness probes. While the startup probe has not yet succeeded, liveness and readiness probes are disabled. Use it for applications with startup times greater than 30 seconds. Set a high failureThreshold (e.g., 60) with a short periodSeconds (e.g., 5) to allow up to 300 seconds for startup. Once the startup probe succeeds, liveness and readiness probes begin their normal checks.

All three probes support the same handler types: httpGet, tcpSocket, exec, and grpc.

# probes.yaml — production configuration for a slow-starting app
apiVersion: v1
kind: Pod
metadata:
  name: probe-demo
spec:
  containers:
  - name: app
    image: myapp:1.0.0
    ports:
    - containerPort: 8080
    startupProbe:
      httpGet:
        path: /startupz
        port: 8080
      failureThreshold: 60
      periodSeconds: 5
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      failureThreshold: 3
      periodSeconds: 15
      initialDelaySeconds: 0
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      failureThreshold: 3
      periodSeconds: 10
      initialDelaySeconds: 0

Liveness/Readiness/Startup Probe Comparison Table

While the previous section explained each probe's purpose, a comparison table helps you quickly decide which probe to use in any scenario. The table below summarizes the differences across key dimensions: behavior on failure, impact on the Pod, typical use cases, handler types, and best practices for configuration.

When to combine: Always pair readiness probes with startup probes for applications that warm up slowly. The startup probe disables liveness and readiness during boot, preventing premature restarts and premature traffic. For fast-starting apps (<30s), a simple readiness probe with initialDelaySeconds may suffice, but adding a startup probe costs nothing and adds safety.

Deployment Strategies: RollingUpdate vs Recreate

When updating a Deployment, Kubernetes supports two strategies: RollingUpdate (default) and Recreate. The choice between them directly impacts availability, resource usage, and rollout speed.

RollingUpdate: Replaces old Pods with new Pods incrementally. Controlled by two parameters: - maxSurge: How many extra Pods (count or percentage) can be created above the desired replica count during the update. - maxUnavailable: How many Pods (count or percentage) can be unavailable during the update.

During a rolling update, the Deployment controller creates new Pods in a new ReplicaSet, waits for them to become ready, then scales down the old ReplicaSet. The process repeats until all old Pods are replaced. This strategy enables zero-downtime deployments, but at the cost of requiring extra cluster capacity (at least maxSurge proportion of overhead).

Recreate: Terminates all old Pods simultaneously, then creates new Pods. This is a simple 'kill all, start all' pattern. During the update, the application is completely unavailable — no traffic can be served until all new Pods are ready. Use this strategy only when the application cannot have multiple versions running concurrently (e.g., stateful applications with incompatible schemas, or when file locks prevent coexistence). Recreate is also useful for cost-constrained environments where you cannot afford the overhead of extra Pods during a rollout.

# rollingupdate-strategy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rolling-deploy
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: rolling
    spec:
      containers:
      - name: app
        image: nginx:1.25
---
# recreate-strategy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: recreate-deploy
spec:
  replicas: 3
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: recreate
    spec:
      containers:
      - name: app
        image: nginx:1.25

RollingUpdate vs Recreate Visual Comparison

While the previous section explained when to use each strategy, this visual comparison highlights the key differences in resource usage, timeline, and availability during a deployment. Use this diagram and table to communicate rollout behavior to your team and to decide which strategy fits your workload.

Timeline Comparison (3 replicas, 10s per Pod startup): - Recreate: 0s — all 3 old pods terminated. 10s — first new pod ready. 20s — all new pods ready. Total downtime: ~10s (from 0s to first pod ready). - RollingUpdate (maxSurge=1, maxUnavailable=0): 0s — 3 old pods serving, 1 new pod created. 10s — new pod ready, old pod terminated. 15s — second new pod created. 25s — second new pod ready, second old pod terminated. 30s — third new pod created. 40s — third new pod ready, third old pod terminated. Total downtime: 0s (always at least 2 pods serving).

Resource Peak Comparison (3 replicas, each request 256Mi memory, 500m CPU): - Recreate: 0s-10s: 0 pods, 0 resources used. 10s-20s: 3 pods, 768Mi/1.5 CPU. Peak: same as steady state. - RollingUpdate (maxSurge=1, maxUnavailable=0): Overlap phase: 2 old + 1 new = 3 pods, plus 1 extra during surge = 4 pods simultaneously. Peak memory: 4×256Mi = 1024Mi. Peak CPU: 4×500m = 2000m. Requires 33% more capacity than steady state.

Resource Management: Requests, Limits, and QoS Classes

Every container in Kubernetes should specify CPU and memory requests and limits. These settings directly affect scheduling, runtime performance, and cluster stability. Misconfiguring them is a top cause of production incidents.

Requests are the minimum amount of resources guaranteed to the container. The Kubernetes scheduler uses requests to make placement decisions — it only schedules a Pod on a node that has at least the sum of all Pod requests available. Requests also ensure the container gets at least that much CPU and memory under contention.

Limits are the maximum amount of resources a container is allowed to consume. If a container exceeds its CPU limit, it gets throttled (not killed). If it exceeds its memory limit, it gets OOMKilled (exit code 137). Limits prevent a single container from starving other containers on the same node.

QoS Classes: Kubernetes categorizes Pods into three Quality of Service classes based on request and limit settings: - Guaranteed: requests == limits for all resources (e.g., cpu: 500m, limits cpu: 500m). These Pods are least likely to be evicted and get the most predictable performance. - Burstable: At least one container has requests < limits for any resource. These Pods get their requested minimum but can burst to their limit if node capacity is available. - BestEffort: No requests or limits set. These Pods receive no guarantees and are the first to be evicted under node pressure. Avoid BestEffort in production.

Production Best Practices: 1. Always set requests and limits for every container. Never leave them unset. 2. Set requests equal to limits for critical workloads to achieve Guaranteed QoS and predictable performance. 3. Base request values on steady-state resource usage observed in production over 7 days. Do not guess. 4. Set memory limits to 1.5x the observed maximum to handle transient spikes without OOMKill. 5. CPU limits are less critical than memory limits, but still set them to prevent noisy neighbors. CPU throttling is better than OOMKill. 6. Use Vertical Pod Autoscaler (VPA) in recommendation mode to generate initial request/limit values, then refine manually.

# resources.yaml — Guaranteed QoS example
apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    resources:
      requests:
        memory: "256Mi"
        cpu: "500m"
      limits:
        memory: "256Mi"
        cpu: "500m"
---
# Burstable example
apiVersion: v1
kind: Pod
metadata:
  name: burstable-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    resources:
      requests:
        memory: "128Mi"
        cpu: "100m"
      limits:
        memory: "512Mi"
        cpu: "1"

Resource Management Best Practices (Requests/Limits)

While the previous section explained the mechanics of requests, limits, and QoS classes, this section provides a concise list of best practices that you can apply immediately to your production Deployments. These recommendations come from real-world incidents and thousands of production clusters.

1. Never leave resources unset. A container without requests runs as BestEffort QoS — it gets no guarantees and is first to be evicted. A container without limits can consume all node memory and OOMKill other Pods. Always set both.

2. Base requests on steady-state usage, not peak. Monitor your application for at least 7 days using kubectl top pod or a metrics system (Prometheus). Set requests to the 50th percentile of observed usage. Set limits to 1.5x the 95th percentile for memory (to handle spikes) and 2x the 95th percentile for CPU (since throttling is acceptable).

3. Use Guaranteed QoS for all critical Deployments. Setting requests == limits gives the Pod the highest eviction priority and most stable CPU scheduling. The only cost is that you cannot overcommit, but for critical services this is a feature, not a bug.

4. Test resource configurations under load. A common mistake: setting memory too low based on idle measurement. During traffic spikes, memory usage can double. Run load tests that mimic peak traffic and verify that memory stays below the limit.

5. Avoid setting identical CPU and memory values for all containers. Each container type has different resource profiles. For example, a sidecar proxy (Envoy) needs different requests than the main application container. Use per-container settings.

6. Implement resource quotas at the namespace level. Even with per-Pod limits, a runaway Deployment can create many Pods that collectively use too many resources. Use ResourceQuota objects to enforce aggregate limits per namespace.

7. Monitor for OOMKill events and CPU throttling. Set up alerts on kube_pod_container_status_terminated_reason (OOMKilled) and container_cpu_cfs_throttled_seconds_total. These events indicate misconfigured resource limits.

8. Use VPA in recommendation mode (not auto) for initial tuning. The Vertical Pod Autoscaler can analyze historical usage and suggest request/limit values. Review its recommendations before applying. Do not enable auto mode for stateless workloads unless you understand the disruption it causes (Pod restart on every recommendation).

9. Consider using alternative scheduling policies. For batch workloads that can tolerate lower priority, use priorityClassName to distinguish critical vs. best-effort. Combined with Guaranteed QoS, priority ensures your most important workloads survive contention.

10. Document your resource strategy. Include typical request/limit values for each service, the reasoning behind them, and instructions for adjusting. This prevents future engineers from guessing or copying incorrect values from other services.

# deployment-with-resources.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-deployment
spec:
  template:
    spec:
      containers:
      - name: app
        image: myapp:1.0
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "768Mi"
            cpu: "1000m"
      - name: sidecar
        image: envoy:latest
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "500m"

Production Operations: Essential kubectl Patterns

Managing Deployments in production requires more than just apply. You need to be able to inspect the rollout history, trigger instant rollbacks, and scale on demand.

# 1. Trigger a manual rolling update by changing the image
kubectl set image deployment/forge-api-deployment api=io.thecodeforge/api:2.1.0

# 2. Watch the rollout in real-time
kubectl rollout status deployment/forge-api-deployment

# 3. Emergency! Roll back to the previous stable version
kubectl rollout undo deployment/forge-api-deployment

# 4. View the rollout history to find a specific revision
kubectl rollout history deployment/forge-api-deployment

# 5. Scale the workload horizontally (manual scaling)
kubectl scale deployment/forge-api-deployment --replicas=10

# 6. Deep inspection of pod failures
kubectl describe pod <pod-name>
kubectl logs -f deployment/forge-api-deployment --all-containers