CrashLoopBackOff: Fix a K8s Pod Stuck Restarting

Nothing ruins your morning like a deploy that looked green in CI and then parks every new pod in CrashLoopBackOff. The Service has no healthy endpoints, the rollout is stuck, and kubectl get pods shows a restart count climbing while the retry delay stretches further apart. Your instinct says the cluster is broken. It isn't. Somewhere in your manifest or startup path there's a small mistake, and the kubelet is patiently retrying it forever.

CrashLoopBackOff is Kubernetes' most-seen and most-misread pod state. Engineers treat it as an error and start changing things at random: bump memory, delete the pod, relax the probe. Sometimes one of those sticks, but nobody learns why, so the same typo kills the next deploy. The growing delay — 10 seconds, then 20, then 40, up to 5 minutes — feels like the cluster giving up. It's protecting the node while it waits for your fix.

This guide gives you a repeatable way out. You'll learn what the backoff is doing, the five causes behind nearly every loop (bad command, missing config, killer probes, OOMKilled, instant exit 0), and how to tell CrashLoopBackOff from Error, OOMKilled, and CreateContainerConfigError. You'll get a crashing-versus-fixed Deployment pair, a debug sequence built on kubectl logs --previous and describe pod, and the habits — startup probes, data-sized resources, Jobs for finite work — that stop the loop coming back.

CrashLoopBackOff Is a Waiting State, Not an Error

When a container dies, the kubelet doesn't give up — it restarts it. Die again, and it waits 10 seconds. Then 20, 40, 80, 160, capping at 300 seconds (5 minutes). That growing pause is the 'BackOff' in CrashLoopBackOff, and it exists to protect the node: without it, a pod crash-looping ten times a second would spam the API server and churn CPU forever. You'll see the state in kubectl get pods while kubectl describe pod shows reason: CrashLoopBackOff, a climbing restartCount, and a lastState.terminated block with the exit code that tells you how the container actually died.

Here's the part most engineers miss: the backoff resets the moment a container stays alive. There's no stuck timer to clear and no node state to flush. Fix the crash cause, and the replacement pod goes Ready on its first try — the hour of accumulated delay doesn't carry over. That's why deleting the pod 'to clear the backoff' never helps: the new pod runs the same broken image and config, crashes the same way, and climbs the same 10s-to-5min ladder. The delay is pure pacing, and pacing is never the problem.

So read the state as a message, not a malfunction. CrashLoopBackOff says: this container starts (or tries to) and dies, over and over, and I'm spacing out retries while you investigate. Your job isn't to fight the scheduler — it's to find the death cause in the previous container's logs and the exit code, fix it in the manifest or image, and roll forward. Everything else in this guide is about doing exactly that as fast as possible.

Bad Command or Args: the One-Line Killer

The single most common crash-loop cause is also the most embarrassing: the container's command doesn't work. Maybe the manifest overrides command with a binary that isn't in the image (python when only python3 exists), passes a flag the app renamed last sprint, or points at a file that moved (python app.py when the code now lives at src/main.py). The container starts, the entrypoint fails instantly with 'executable not found' or a usage error, and you get exit code 1 or 2 within a second. Three of those in a row and you're in backoff.

Compare the pair below. The broken Deployment passes --port=8080, which argparse rejects because the app now expects --listen-port — the container exits code 2 two seconds after start, every time. The fixed Deployment passes the flag the image actually accepts. Nothing else changed: same image, same resources, same probes. That's typical — crash loops from bad commands are one-line fixes once you see the usage error in kubectl logs <pod> --previous.

Build the habit that prevents this class entirely: run the exact command locally with docker run before it ever reaches the cluster, and add a CI smoke step that boots the image with --help or a dry-run flag. If the entrypoint is wrong, CI should go red — not your checkout Service at noon. And when you inherit someone else's image, inspect its real entrypoint (docker inspect or the Dockerfile CMD) instead of guessing what the binary is called.

# BROKEN: passes a flag the image no longer accepts -> exit code 2 in ~2s
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  labels:
    app: checkout
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: registry.example.com/shop/checkout:2.4.1
          args: ["--serve", "--port=8080"]   # stale flag: app wants --listen-port
          ports:
            - containerPort: 8080
---
# FIXED: identical except the flag matches the image's real interface
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  labels:
    app: checkout
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: registry.example.com/shop/checkout:2.4.1
          args: ["--serve", "--listen-port=8080"]  # matches argparse interface
          ports:
            - containerPort: 8080

Missing Env Vars and Config That Kill Startup

The second classic is the app that boots, looks for a setting, doesn't find it, and quits. A Python service throws KeyError: 'STRIPE_KEY', a Go binary logs 'required env DATABASE_URL not set' and exits 1, a Java app can't reach postgres because DB_HOST came from a ConfigMap in the wrong namespace. The container technically runs — it just commits suicide within seconds because its world is incomplete. kubectl logs --previous shows the complaint plainly, usually in the last five lines before the traceback.

Note the sharp edge here: a manifest that references a ConfigMap or Secret that doesn't exist at all fails differently — Kubernetes won't even start the container and you'll see CreateContainerConfigError instead. CrashLoopBackOff from config means the reference resolved but the content disappointed: a typo'd key (datebase_url), a value the app can't parse, or a Secret that exists but holds last quarter's rotated password. The container started, read its config, hated it, and left.

The fixed manifest below wires config explicitly: required values come from a Secret and a ConfigMap that live in the same namespace, with names spelled exactly as created. Pair that with fail-fast startup code — validate every required variable on boot and log its absence by name — and the next missing setting announces itself in one log line instead of a bare traceback. Validate references before applying: a quick kubectl get configmap,secret plus a dry-run render catches the typo'd key while it's still cheap.

# FIXED: explicit env wiring — Secret + ConfigMap in the SAME namespace as the pod
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
  namespace: production
  labels:
    app: payments
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payments
  template:
    metadata:
      labels:
        app: payments
    spec:
      containers:
        - name: payments
          image: registry.example.com/shop/payments:1.9.0
          ports:
            - containerPort: 8080
          env:
            - name: STRIPE_KEY               # required: app fails fast naming it if absent
              valueFrom:
                secretKeyRef:
                  name: payments-secrets     # must exist in namespace: production
                  key: stripe-key
            - name: DB_HOST
              valueFrom:
                configMapKeyRef:
                  name: payments-config      # must exist in namespace: production
                  key: db-host
# Pre-apply check (same namespace!):
# kubectl get secret payments-secrets -n production
# kubectl get configmap payments-config -n production

Liveness Probes That Kill Healthy Apps

Liveness probes restart containers — that's their entire job. So when a liveness probe is misconfigured, it manufactures a crash loop out of a perfectly healthy app. The usual story: initialDelaySeconds is 5 but the app needs 45 seconds to warm its cache (longer under CPU-throttled morning load), so the kubelet starts probing a half-booted process, gets connection refused three times, and SIGTERMs a container that was 10 seconds from Ready. The app log shows a normal startup chopped off mid-line. Describe shows 'Liveness probe failed' events marching in step with the restarts.

Keep the two probe types straight and this class disappears. Readiness controls traffic: fail it and the pod leaves the Service endpoints, no restart. Liveness controls life: fail it past failureThreshold and the kubelet kills the container, which under restartPolicy: Always means a restart — and repeated restarts mean backoff. Never gate boot-time on liveness. The manifest below does it right: a startupProbe owns the boot window (up to 60 checks × 5s = 5 minutes of grace), liveness only starts after startup succeeds, and readiness stays independent so a slow dependency removes the pod from traffic instead of killing it.

Size every number from measurement, not hope. Time your worst-case cold start under load (p99, not the laptop average), then set startup failureThreshold above it. A probe tuned on a developer's laptop with a warm image cache will murder the same app on a cold node at 9 AM — and you'll spend the morning debugging an app that was never broken.

# FIXED: startup probe owns the boot window; liveness only judges steady state
apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog
  labels:
    app: catalog
spec:
  replicas: 3
  selector:
    matchLabels:
      app: catalog
  template:
    metadata:
      labels:
        app: catalog
    spec:
      containers:
        - name: catalog
          image: registry.example.com/shop/catalog:3.2.0
          ports:
            - containerPort: 8080
          startupProbe:                        # owns boot: 60 x 5s = 5 min grace
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 5
            failureThreshold: 60
          livenessProbe:                       # starts only after startup succeeds
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 15
            failureThreshold: 3
          readinessProbe:                      # traffic only, never restarts
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 10
            failureThreshold: 2

Exit Codes and Lookalikes: 137, Error, OOMKilled, Config Errors

Exit codes are the fastest triage tool you have, so learn the four you'll actually meet. Exit 1 (or 2) is your app raising its hand: an unhandled exception, a bad flag, a failed assertion — the answer is in the --previous logs. Exit 137 (128 + 9, SIGKILL) is the kernel OOM-killing your container for breaching limits.memory: the process vanishes mid-request with no error because it never got to log one. Exit 0 is success — which under restartPolicy: Always still triggers a restart, looping forever. And 'no exit code at all' plus CreateContainerConfigError means the container never started: bad volume mount, missing ConfigMap, or an env reference to an object that doesn't exist.

The lookalikes matter because each sends you to a different layer. Error (without backoff) is just the opening act — the first crashes before the delay ladder starts; debug it identically. OOMKilled is a resource verdict, not an app verdict: confirm with the reason field and memory metrics, then raise limits.memory above peak plus headroom or fix the leak. CreateContainerConfigError is a manifest verdict: kubectl describe pod names the missing object ('configmap "x" not found'), and the fix is creating it or correcting the reference — no amount of log-reading helps because there's no container to have logged.

The command block below pulls all of this in three queries: exit codes and reasons straight from the pod status, the events that separate probe kills from instant crashes, and live memory against the limit. Run them in order and you'll know which of the four you're holding within a minute — then jump to that section's fix instead of guessing.

# 1) Exit codes + reasons: the 10-second triage (137 = OOM, 1/2 = app, 0 = finished)
kubectl get pod checkout-7d9c8f6b4c-x2v4p -o jsonpath='{range .status.containerStatuses[*]}{.name} restarts={.restartCount} lastExit={.lastState.terminated.exitCode} reason={.lastState.terminated.reason}{"\n"}{end}'

# 2) Events: probe kills vs instant crashes vs missing objects
kubectl describe pod checkout-7d9c8f6b4c-x2v4p | grep -E 'Reason|Message|probe|Back-off|OOM|not found' | tail -20

# 3) Memory vs limit: confirm the OOM story before raising anything
kubectl top pod checkout-7d9c8f6b4c-x2v4p --containers
kubectl get pod checkout-7d9c8f6b4c-x2v4p -o jsonpath='{range .spec.containers[*]}{.name} limits={.resources.limits.memory}{"\n"}{end}'

The Debug Sequence: From Symptom to Fixed Deploy

When the pager goes off, run this sequence exactly — it resolves nearly every crash loop in under ten minutes. First, kubectl logs <pod> --previous: the dead container's last words, where usage errors and tracebacks live. Second, kubectl describe pod: exit codes, restart count, and the event stream that separates probe kills ('Liveness probe failed') from instant deaths from missing objects. Third, the exit-code query from the previous section to name your category: 1/2, 137, 0, or never-started. By step three you know which fix to apply, and you've spent maybe three minutes.

Fourth, fix forward in the manifest — never hand-edit the crashing pod, which a controller will replace or which teaches you nothing reproducible. Bad flag? Correct args and apply. Missing config? Create the object in the right namespace and roll out. Probe killing boot? Add the startup probe. OOM? Raise limits.memory with headroom. Fifth, verify with kubectl rollout status plus kubectl get events scoped to the new pods, and confirm restart counts stay at zero for two full minutes. A pod that survives one backoff window (5 minutes worst case) is fixed; one that dies at 90 seconds will fool a 60-second glance.

Make the loop structurally unlikely to return. Gate rollouts with readiness probes and maxUnavailable: 1 so a bad revision can't empty the Service. Run finite work in Jobs, not Deployments. Smoke-test the image's real command in CI. These five habits don't just fix today's loop — they delete the whole category of 2 AM pages where the cluster retries your typo forever while you sleep.

# The 5-step crash-loop sequence — run in order, under 10 minutes
# 1) Why did it die? (dead container's last words)
kubectl logs checkout-7d9c8f6b4c-x2v4p --previous --tail=50

# 2) Context: exit codes, restart count, event stream
kubectl describe pod checkout-7d9c8f6b4c-x2v4p | tail -40

# 3) Name the category, then fix FORWARD in the manifest (never hand-edit pods)
#    exit 1/2 -> fix command/args or app bug
#    exit 137 -> raise limits.memory
#    exit 0   -> move workload to a Job
kubectl apply -f checkout-deployment-fixed.yaml
kubectl rollout status deploy/checkout --timeout=180s

# 4) Verify: new pods stay at zero restarts for 2+ minutes
kubectl get pods -l app=checkout -w
kubectl get events --field-selector involvedObject.kind=Pod -w | grep -iE 'fail|kill|oom|back-off'