Docker Troubleshooting Guide: Diagnose and Fix Container Crashes in Production

Your container just died at 3 AM. No alert, just a 137 exit code. Now what? Most devs panic and restart — then it dies again. That's not troubleshooting, that's gambling.

The problem is that Docker abstracts away the OS, but it doesn't abstract away failures. Containers crash for the same reasons processes crash: out of memory, disk full, deadlocks, or misconfigured limits. The difference is you have fewer tools inside the container to debug.

By the end of this guide, you'll be able to diagnose any container crash in under 5 minutes. You'll know exactly which command to run, what to look for, and how to fix it permanently. No more guessing, no more restart loops.

Why Containers Crash: The Three Horsemen

Before you run any command, understand the three root causes of container failure: resource exhaustion (OOM, disk full), application bugs (segfaults, uncaught exceptions), and infrastructure issues (Docker daemon restart, network timeouts). Each leaves a different signature.

OOM kills leave exit code 137 and a kernel log. Disk full leaves exit code 1 with a 'no space left on device' error in logs. Segfaults leave exit code 139. Infrastructure issues often leave exit code 125 (daemon error) or 126 (command not executable).

Your first step is always: check the exit code. docker inspect --format '{{.State.ExitCode}}'. Then match it to the list above. This narrows your search from everything to one category.

#!/bin/bash
# io.thecodeforge — DevOps tutorial
# Check exit code of a stopped container
docker inspect my_app --format '{{.State.ExitCode}}'
# Output: 137
# Then check OOM logs
dmesg | grep -i oom | tail -5
# Output: [12345.678] oom-kill: constraint=CONSTRAINT_MEMCG ...

Memory Troubleshooting: Beyond the Heap

Memory issues are the #1 cause of container crashes in production. But most devs only look at the app heap. That's like checking your car's fuel gauge when the engine seizes — you're missing the oil pressure.

Inside a container, memory is used by: the app heap, native code (JNI, C libraries), thread stacks, Metaspace (JVM), page cache, and tmpfs mounts. All of it counts against the memory limit. If the sum exceeds the limit, the kernel kills the container.

To see real memory usage: docker stats <container>. Look at the MEM USAGE / LIMIT column. If it's close to 100%, you're on the edge. Then use docker exec <container> cat /sys/fs/cgroup/memory/memory.usage_in_bytes to get the exact cgroup value.

For JVM apps, add -XX:NativeMemoryTracking=summary and use jcmd <pid> VM.native_memory to see non-heap usage. For Node.js, check process.memoryUsage().

#!/bin/bash
# io.thecodeforge — DevOps tutorial
# Check memory usage inside container
docker stats my_app --no-stream
# Output:
# CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %
# abc123         my_app    45.2%     3.8GiB / 4GiB       95%
# Check cgroup memory
docker exec my_app cat /sys/fs/cgroup/memory/memory.usage_in_bytes
# Output: 4080218931 (about 3.8GiB)
# Check OOM kills for this container
dmesg | grep -i 'my_app' | grep -i oom
# Output: (if any OOM kill happened)

Disk Full: The Silent Killer

A full disk doesn't crash your container — it freezes it. Writes fail silently, logs stop, and health checks time out. Then your orchestrator kills the container for being unhealthy. The real fix isn't restarting; it's freeing space.

First, check disk usage on the host: df -h. If any mount is at 100%, that's your problem. Docker stores images, containers, and volumes under /var/lib/docker by default. Use docker system df to see how much space each component uses.

Common culprits: unused images (docker image prune -a), stopped containers (docker container prune), and dangling volumes (docker volume prune). For production, set up a cron job to run docker system prune -f --volumes weekly.

But sometimes the issue is inside the container — logs growing unbounded. Always mount a log directory to a dedicated volume with a size limit, or use a log shipper that rotates.

#!/bin/bash
# io.thecodeforge — DevOps tutorial
# Check disk usage on host
df -h /var/lib/docker
# Output:
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/sda1        50G   48G  2.0G  96% /var/lib/docker
# Check Docker disk usage
docker system df
# Output:
# TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
# Images          10        5         20GB      10GB (50%)
# Containers      8         3         5GB       2GB (40%)
# Local Volumes   5         2         10GB      8GB (80%)
# Build Cache     0         0         0B        0B
# Prune unused resources
docker system prune -a --volumes -f

Zombie Processes and PID 1 Problems

Containers run with PID 1 — the init process. If your app doesn't handle signal forwarding and reaping zombie children, you get a mess. Zombie processes accumulate, PID 1 doesn't respond to SIGTERM, and your container becomes unkillable.

This is a classic rookie mistake: running a Java app directly as java -jar app.jar. Java doesn't reap zombies. If your app spawns child processes (e.g., forking a shell command), those children become zombies when they exit, because PID 1 (Java) doesn't call wait().

The fix: use a lightweight init system like tini or dumb-init. Add it to your Dockerfile: ENTRYPOINT ["/usr/bin/tini", "--"] then CMD ["java", "-jar", "app.jar"]. Tini handles signal forwarding and reaping.

To check for zombies inside a container: docker exec ps aux | grep 'Z'. If you see zombie processes, you need an init system.

# io.thecodeforge — DevOps tutorial
FROM openjdk:11-jre-slim
# Install tini
RUN apt-get update && apt-get install -y tini && rm -rf /var/lib/apt/lists/*
# Use tini as init
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["java", "-jar", "app.jar"]

Network Timeouts and DNS Resolution

Your container can't reach the database. But it worked yesterday. Network issues in Docker are often DNS-related. By default, containers use the host's DNS via /etc/resolv.conf. If that file is stale or the DNS server is unreachable, name resolution fails.

First, test DNS inside the container: docker exec <container> nslookup db.example.com. If it fails, check the container's /etc/resolv.conf: docker exec <container> cat /etc/resolv.conf. It should list valid nameservers.

Common fix: set --dns flag to a reliable DNS server like 8.8.8.8. Or use --dns-search to add search domains. For production, run a local DNS cache like dnsmasq in a sidecar container.

Another gotcha: Docker's default bridge network doesn't allow DNS resolution by container name. Use a user-defined network: docker network create mynet then docker run --network mynet. This enables DNS resolution using container names as hostnames.

#!/bin/bash
# io.thecodeforge — DevOps tutorial
# Test DNS inside container
docker exec my_app nslookup db.example.com
# Output:
# Server:         8.8.8.8
# Address:        8.8.8.8#53
# Non-authoritative answer:
# Name:   db.example.com
# Address: 10.0.0.1
# Check resolv.conf
docker exec my_app cat /etc/resolv.conf
# Output:
# nameserver 8.8.8.8
# nameserver 8.8.4.4

Health Checks: The First Line of Defense

You shouldn't wait for a crash to know something's wrong. Health checks let Docker monitor your container and restart it automatically if it becomes unhealthy. But most people write health checks that are too simple — they check if the process is running, not if the app is actually working.

A good health check tests a real endpoint: curl -f http://localhost:8080/health. It should return HTTP 200 within a timeout. Set --health-interval=30s, --health-retries=3, and --health-start-period=30s to give the app time to start.

In your Dockerfile: HEALTHCHECK --interval=30s --timeout=3s --start-period=30s --retries=3 CMD curl -f http://localhost:8080/health || exit 1.

To see health status: docker inspect --format '{{.State.Health.Status}}' . It returns healthy, unhealthy, or starting. If it's unhealthy, check the last 5 health check logs: docker inspect --format '{{json .State.Health.Log}}' .

# io.thecodeforge — DevOps tutorial
FROM nginx:alpine
# Install curl for health check
RUN apk add --no-cache curl
# Add health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=30s --retries=3 \
  CMD curl -f http://localhost/ || exit 1
EXPOSE 80

cgroups v2 Specific Error Catalog: 8+ Errors and Exact Fix Paths

Modern Linux distributions (Ubuntu 22.04+, Fedora 35+, RHEL 9+) use cgroups v2 by default. Docker's cgroups v2 support has matured, but several errors still bite engineers. Catalog:

1. cgroup2: unknown option — Docker < 20.10 on cgroups v2. Fix: upgrade Docker to 20.10+.
2. permission denied writing to /sys/fs/cgroup — container lacks CAP_SYS_ADMIN or cgroupns mode restricts writes. Fix: docker run --privileged or --security-opt systempaths=unconfined.
3. write /sys/fs/cgroup/...: device or resource busy — cgroup fs already mounted with different options. Fix: ensure systemd.unified_cgroup_hierarchy=1 on kernel cmdline.
4. containerd: failed to mount cgroup2 — containerd < 1.5 incompatible with kernel cgroup2. Fix: apt-get install --only-upgrade containerd.io.
5. runc: symlink /sys/fs/cgroup — runc < 1.0.0-rc93. Fix: upgrade containerd which bundles runc.
6. OCI runtime create failed: unable to get device controller — missing memory or io controller. Fix: enable CONFIG_CGROUP_MEMORY and CONFIG_BLK_CGROUP in kernel.
7. failed to write "max" to memory.max — swap limit requires memory.swap.max which may be absent. Fix: set --memory-swap equal to --memory or configure kernel with CONFIG_MEMCG_SWAP=y.
8. cgroup: fork failed: resource temporarily unavailable — cgroup pids limit hit. Fix: increase --pids-limit (default: unlimited in Docker, but systemd may set 4096).

Always check docker info | grep cgroup to see which cgroup version Docker is using. For mixed cgroups v1/v2 hosts, set "cgroupns": "host" in daemon.json.

#!/bin/bash
# io.thecodeforge — DevOps tutorial

# Check which cgroup version Docker is using
docker info | grep -i cgroup
# Output: Cgroup Driver: systemd
#         Cgroup Version: 2

# Check kernel cgroup version
stat -fc %T /sys/fs/cgroup/
# Output: cgroup2fs (v2) or tmpfs (v1)

# List available cgroup controllers
cat /sys/fs/cgroup/cgroup.controllers
# Output: cpu io memory pids

# Fix common error: set daemon.json for cgroupv2 compatibility
echo '{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "cgroupns": "host"
}' > /etc/docker/daemon.json

Healthcheck Failure Debugging: Why Your Service Is Stuck in 'starting'

Docker health checks can fail in subtle ways. The most common: a service stuck in starting status indefinitely. This happens when --health-start-period is too short — the health check command runs before the app is ready, fails, and Docker keeps retrying until start_period elapses. After that, if the first check still fails, the container goes unhealthy. Symptoms: docker inspect --format '{{.State.Health.Status}}' shows starting for longer than expected. Debug with docker inspect --format '{{json .State.Health.Log}}' to see the last 5 health check outputs. If the health check command itself is failing (e.g., curl not found), install it in the Dockerfile or use a shell-based check like CMD pgrep nginx || exit 1. Cascading healthcheck failures: when service A depends on service B, and B's health check fails, A's health check may also fail (e.g., A checks B's endpoint). Break circular dependencies by making health checks independent. Use start_period generously (60s for Spring Boot, 30s for Node.js) and set retries to 3 with interval of 15-30s.

#!/bin/bash
# io.thecodeforge — DevOps tutorial

# Check health status
docker inspect --format '{{.State.Health.Status}}' my_container
# Output: starting (if stuck, check logs below)

# View health check log history
docker inspect --format '{{json .State.Health.Log}}' my_container | jq .
# Output:
# [
#   {
#     "Start": "2026-07-11T10:00:00Z",
#     "End": "2026-07-11T10:00:01Z",
#     "ExitCode": 1,
#     "Output": "curl: (7) Failed to connect to localhost port 8080"
#   }
# ]

# Fix: increase start period and verify command exists
docker run --health-cmd="curl -f http://localhost:8080/health || exit 1" \
  --health-interval=15s \
  --health-retries=5 \
  --health-start-period=60s \
  --health-timeout=5s \
  my_app

Exit Code Reference Table: What Each Code Means and How to Respond

Docker exit codes are your first diagnostic signal. Here is the comprehensive reference:

Pro tip: Use docker inspect --format 'ExitCode={{.State.ExitCode}} OOM={{.State.OOMKilled}}' to quickly see both exit code and whether OOM was the cause.

#!/bin/bash
# io.thecodeforge — DevOps tutorial

# Quick exit code + OOM status
docker inspect my_container --format 'ExitCode={{.State.ExitCode}}  OOM={{.State.OOMKilled}}'
# Output: ExitCode=137  OOM=true

# Check all exit codes for recent restarts
docker inspect my_container --format '{{range .State.Health.Log}}{{.ExitCode}}{{"\n"}}{{end}}'

# List all stopped containers with exit codes
docker ps -a --format 'table {{.Names}}\t{{.Status}}'

PID 1 Signal Handling: Why Your Container Ignores SIGTERM and Requests Get Dropped

When you run docker stop, Docker sends SIGTERM to PID 1 inside the container. If PID 1 doesn't handle SIGTERM, the default behavior is to ignore it. After 10 seconds, Docker sends SIGKILL. This causes delayed deploys (orchestrator waits for the stop timeout), dropped in-flight requests (SIGKILL doesn't allow cleanup), and zombie processes (no reaper). The root cause: running the application directly as PID 1 (e.g., CMD ["java", "-jar", "app.jar"]). Java, Node.js, Python, and Go default binaries don't reap zombies or forward signals. The solution: use a lightweight init system like tini or dumb-init. In Dockerfile: ENTRYPOINT ["/usr/bin/tini", "--"] and CMD ["your-app"]. Tini ensures SIGTERM is forwarded to the child process and waits for it to exit before allowing the container to die. It also reaps zombie processes. For Node.js apps specifically, the --init flag on docker run achieves the same: docker run --init node:18-alpine node app.js. This inserts tini automatically. Verify: docker exec ps aux — PID 1 should be tini, your app should be PID 2 or higher.

# io.thecodeforge — DevOps tutorial

# Option 1: Dockerfile with tini
FROM node:18-alpine
RUN apk add --no-cache tini
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["node", "server.js"]

# Option 2: Simpler — use docker run --init
# docker run --init -d --name myapp node:18-alpine node server.js

# Verify PID 1
# $ docker exec myapp ps aux
# PID   USER     TIME  COMMAND
#   1   root     0:00 /sbin/tini -- node server.js
#   7   root     0:00 node server.js
#  14   root     0:00 ps aux