ImagePullBackOff: Fix K8s Image Pull Failures

You push the deploy, watch the rollout — and every new pod parks in ImagePullBackOff. The app code is fine. The manifest passed review. But nothing starts, because the kubelet can't download the container image: a typo'd tag, a private registry with no credentials, a token that expired Friday night, or a rate limit burned at 9 AM scale-up. Your cluster is healthy and your image may be perfect — the two just can't meet.

ImagePullBackOff is the pull-side cousin of CrashLoopBackOff, and confusing the two wastes incident time. In a crash loop the container starts and dies; here it never starts at all. There are no logs — kubectl logs just says nothing is running. The evidence lives in pod events ('Failed to pull image' plus the registry's reason) and a manual pull test. Engineers who reach for app logs first lose twenty minutes; engineers who run kubectl describe pod first find the answer in thirty seconds.

This guide covers the five pull failures you'll meet: wrong name/tag, the mutable latest trap, private registries without imagePullSecrets, expired credentials, and rate limits or blocked egress. You'll get the exact docker-registry Secret command, a fixed Deployment plus imagePullSecrets manifest, and the habits — digest pinning, internal mirrors, CI pull checks — that make image pulls boring again.

ImagePullBackOff Is a Waiting State, Not a Failure

ImagePullBackOff is a waiting state owned by the kubelet, and the name describes the whole mechanism: the kubelet tried to pull your container image, the pull failed, and now it's backing off between retries. First failure shows as ErrImagePull; after repeated failures the growing retry delay earns the BackOff suffix. The delay stretches from seconds to minutes, which is why a pod can sit untouched for a long while and then suddenly try again. Nothing about your app is involved — no container has ever started, no code has ever run, and no log exists anywhere on the cluster.

That last point is the one that costs teams their first twenty minutes. kubectl logs on an ImagePullBackOff pod returns 'container not running' because there is no container. The evidence lives in exactly two places: the pod's event stream (kubectl describe pod, look for 'Failed to pull image' plus the registry's reason string) and a manual pull test from a machine with equivalent network access (docker pull or crictl pull of the exact image string). The registry's reason is remarkably honest — 'not found', 'unauthorized', 'toomanyrequests' each map to one fix — so read it literally before theorizing.

Treat the state as a message from the kubelet: 'I can't fetch what you asked for, here's why, and I'll keep retrying with backoff while you sort it out.' Your job is to reproduce the fetch failure outside the cluster, fix the reference or the credentials, and roll out fresh pods. Fix the pickup, not the apartment — the cluster is fine.

# 1) Read the registry's reason straight from pod events (not logs — none exist)
kubectl describe pod api-6f7c9d4b8c-q9w2x | grep -A 3 -iE 'Failed to pull image|Failed to pull'

# 2) Reproduce the kubelet's failure from outside the cluster
#    Use the EXACT string from the event above:
docker pull registry.example.com/shop/api:2.4.1
# ...or from a node / debug pod when laptop network differs:
# crictl pull registry.example.com/shop/api:2.4.1

# 3) Map the reason to the fix:
#    'not found' / 'manifest unknown' -> typo in name/tag (Cause 1)
#    'unauthorized'                   -> missing imagePullSecrets (Cause 3)
#    401/403 after months of success  -> expired credentials (Cause 4)
#    'toomanyrequests' / timeout      -> rate limit or blocked egress (Cause 5)

Cause 1: Wrong Image Name, Tag, or Registry Host

The most common pull failure is also the smallest: the image string is wrong. Registries are literal machines — shop-io vs shop.io, api vs api-v2, v1.2 vs 1.2 vs 1.2.0 are all different universes, and the registry answers 'repository does not exist' or 'manifest unknown' without telling you which character is off. YAML adds its own traps: unquoted strings with special characters, a tag that swallowed a trailing space, or an image field assembled from two Helm values that join badly. One wrong character parks every new pod in the Deployment.

Confirm it in under a minute. kubectl describe pod names the exact string the kubelet tried, and the registry UI (or crane manifest / docker manifest inspect) tells you which tags truly exist. If the string matches a real tag but pulls still fail, you've ruled out typos and the problem is credentials or network — move to those sections instead of re-reading the spelling. The manual pull test is the decider: if docker pull of the exact string fails on your laptop too, it's the reference, not the cluster.

Prevention is a CI job, not discipline. Add a pipeline step that pulls — or at least resolves — every image referenced in your manifests before deploy, and copy image strings from the registry UI instead of typing them from memory. Teams that hand-type registry paths relearn this lesson quarterly; teams with a pull-check gate never ship a typo'd tag twice.

# 1) Show exactly what the kubelet tried to pull
kubectl get pod api-6f7c9d4b8c-q9w2x -o jsonpath='{range .spec.containers[*]}{.name} -> {.image}{"\n"}{end}'

# 2) List tags that REALLY exist (copy the correct string from here, not memory)
#    Docker Hub / Harbor UI, or from the CLI:
docker manifest inspect registry.example.com/shop/api:2.4.1 | head -20
# crane (no daemon needed): crane ls registry.example.com/shop/api | grep 2.4

# 3) The decider: pull the exact string. Fails here = bad reference, not the cluster
docker pull registry.example.com/shop/api:2.4.1

Cause 2: the Mutable latest Tag and Its Surprises

Tags like latest feel convenient and act like traps. A tag is a moving pointer, not a version: every push to latest repoints it at new bytes, and each node resolves the pointer at whatever moment it happens to pull. Push Monday, scale up Tuesday, and Tuesday's nodes run different code from Monday's — same manifest, same tag, different digests, different behavior. Worse, 'rolling back' by reapplying the manifest often changes nothing, because latest still points at the broken bytes. Debugging this is miserable: logs differ per node, and nobody can say which code is actually running.

Confirm the drift by comparing what each pod actually runs. The pod status records the resolved digest per container (imageID), so two pods on the same tag with different imageIDs prove the split. That single query ends all debate about whether 'the deploy went out' — under latest, 'the deploy' is a different artifact per node. The fix is to stop deploying pointers: pin production to an immutable digest (image@sha256:...) or at minimum a fixed semver tag, roll out so every node converges on identical bytes, and record the digest in Git next to the manifest.

Note the pull-policy interaction, because it bites both directions. imagePullPolicy: Always re-resolves the tag on every pod start (good for dev, churny for prod), while IfNotPresent trusts the node's cache (fast, but a re-pushed tag won't refresh cached nodes). With pinned digests the distinction barely matters — identical bytes are identical bytes — which is exactly why digests end the whole class of problem instead of tuning it.

Cause 3: Private Registries Without imagePullSecrets

Private registries answer anonymous pulls with 'unauthorized: authentication required', and the kubelet needs credentials handed to it explicitly — it can't prompt, can't use your laptop's docker login, and can't guess. The handoff mechanism is a docker-registry Secret referenced from the pod spec's imagePullSecrets field (or attached to the namespace's service account so every pod inherits it). Miss any link in that chain and pulls fail: no Secret, Secret in the wrong namespace (Secrets never cross namespaces), or a Secret that exists but no pod references. Each variant produces the identical 'unauthorized' event, which is why engineers insist 'but the Secret exists' while the kubelet keeps failing.

The block below is the complete, working chain. First command mints the Secret from your registry credentials in the workload's namespace — note the -n flag, because default-namespace Secrets don't help production-namespace pods. Second manifest attaches it via imagePullSecrets and pins a digest so the pull is both authenticated and immutable. Verify both links after applying: kubectl get secret confirms the object, and a --previous-free describe of a fresh pod confirms the pull succeeded. If it still says unauthorized, compare the Secret's server string against the image's registry host character by character — that mismatch is the next most common link to break.

For clusters with many private Deployments, stop repeating imagePullSecrets per pod: patch it onto the default service account once per namespace and every pod inherits it. And where your cloud offers short-lived pull helpers — ECR credential helpers, ACR and GAR workload identity — prefer them over long-lived passwords, because a password in a Secret is a rotation ticket you'll eventually forget to buy.

# 0) Mint the Secret first (one per namespace — Secrets never cross namespaces):
# kubectl create secret docker-registry regcred \
#   --docker-server=registry.example.com --docker-username=deploy-bot \
#   --docker-password="$REGISTRY_TOKEN" --docker-email=devops@example.com \
#   -n production
---
# FIXED: private image + imagePullSecrets + pinned digest
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: production
  labels:
    app: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      imagePullSecrets:               # kubelet sends these creds to the registry
        - name: regcred               # must exist in namespace: production
      containers:
        - name: api
          image: registry.example.com/shop/api@sha256:9f3c4a1b2e8d7c6b5a4938271605f4e3d2c1b0a9f8e7d6c5b4a39281706f5e4d3
          ports:
            - containerPort: 8080

Cause 4: Expired or Revoked Registry Credentials

Credentials die on schedules nobody remembers. Registry tokens, service-account keys, and basic-auth passwords all carry lifetimes — ECR tokens last 12 hours, many cloud keys 90 days, Docker Hub tokens until revoked — and Kubernetes stores a copy of the credential at Secret-creation time, not a live reference. So rotation at the provider silently invalidates every Secret minted from the old value, while the manifest still looks perfect. Old pods keep running (they pulled long ago), which hides the rot until the next rollout, scale-up, or node replacement suddenly needs fresh pulls everywhere and gets 401s instead.

Diagnose by age, not by content. If pulls worked for months and now fail unauthorized with zero manifest changes, ask when the credential was born, not what's in the Secret. Compare against the provider's lifetime for that credential type; a 91-day-old key against a 90-day policy is a confession. The fix is rotation in the right order: mint the new credential at the provider, recreate the Secret with the same name (delete and re-create, since password fields are awkward to patch by hand), then kubectl rollout restart the affected Deployments so every pod pulls fresh — pods created before rotation still reference resolved images, but any new pull uses the new Secret.

Kill the class with automation and alarms. Prefer credential helpers that mint short-lived tokens on demand (ECR helpers, workload identity on ACR/GAR) so there's nothing to expire. Where long-lived Secrets remain, calendar their rotation at 80 percent of lifetime and alert on image-pull failure events — the event stream reports the first 401 days before users notice, but only if someone is watching it.

# 1) Confirm the age story: when was the Secret born vs the provider lifetime?
#    (ECR tokens: 12h; many cloud keys: 90 days; Hub tokens: until revoked)
kubectl get secret regcred -n production -o jsonpath='{.metadata.creationTimestamp}{"\n"}'

# 2) Mint the new credential at the provider first, then recreate the Secret
#    (delete + re-create keeps the name stable for all referencing pods)
kubectl delete secret regcred -n production
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=deploy-bot \
  --docker-password="$NEW_REGISTRY_TOKEN" \
  --docker-email=devops@example.com \
  -n production

# 3) Force every pod to pull fresh with the new Secret
kubectl rollout restart deploy/api -n production
kubectl rollout status deploy/api -n production --timeout=180s

Cause 5: Rate Limits, Quotas, and Blocked Egress

Even correct credentials fail when the registry won't serve you. Docker Hub allows 100 pulls per 6 hours anonymously and 200 for free authenticated accounts, counted per source IP — and a fleet behind one NAT gateway shares a single IP, so 40 nodes pulling 12 base images each burns 480 pulls against a 100-pull quota in minutes. The registry answers 'toomanyrequests', the kubelet backs off, and morning scale-up stalls with every new pod waiting. Egress policies bite the same way: a NetworkPolicy or firewall that blocks node-to-registry traffic produces timeouts instead of quota errors, but the pod state is identical.

Separate the two causes with one test. If docker pull works from your laptop but fails from a node (ssh in, or run a debug pod and crictl pull), the path is blocked or the quota is IP-bound — laptop success proves the reference and credentials are fine. toomanyrequests names the quota; dial timeouts and connection-refused name the network. Short-term relief is authenticated pulls plus time (the quota window resets), but the durable fix is a pull-through mirror inside your network: Harbor, ECR, GAR, or ACR caching the dozen base images your fleet actually uses, with manifests pointed at the mirror. Cached pulls never touch the public quota and survive a public-registry outage outright.

Size the decision with numbers: count distinct external images in your manifests (most fleets use fewer than 15 base images), multiply by node count and daily pod churn, and compare against your quota tier. Almost every team finds the mirror pays for itself the first time a 9 AM scale-up stops competing with the rest of the internet for pulls.

# 1) Name it: quota or network? Pull from a node, not your laptop
#    (laptop success + node failure = IP-bound quota or blocked egress)
kubectl debug node/$(kubectl get pods -o jsonpath='{.items[0].spec.nodeName}' -l app=api) -it --image=busybox -- crictl pull registry.example.com/shop/api:2.4.1
# Watch for: 'toomanyrequests' (quota) vs 'dial timeout' / 'connection refused' (egress)

# 2) Count your exposure: distinct external images x nodes x daily churn
kubectl get deploy -A -o jsonpath='{range .items[*].spec.template.spec.containers[*]}{.image}{"\n"}{end}' | sort -u

# 3) Durable fix: mirror the base images you actually use, repoint manifests
#    Harbor / ECR / GAR / ACR pull-through cache, then pin:
#    image: mirror.internal/shop/api@sha256:<digest>  (zero public pulls)