Docker Troubleshooting Guide: Diagnose and Fix Container Crashes in Production

Your container just died at 3 AM. No alert, just a 137 exit code. Now what? Most devs panic and restart — then it dies again. That's not troubleshooting, that's gambling.

The problem is that Docker abstracts away the OS, but it doesn't abstract away failures. Containers crash for the same reasons processes crash: out of memory, disk full, deadlocks, or misconfigured limits. The difference is you have fewer tools inside the container to debug.

By the end of this guide, you'll be able to diagnose any container crash in under 5 minutes. You'll know exactly which command to run, what to look for, and how to fix it permanently. No more guessing, no more restart loops.

Why Containers Crash: The Three Horsemen

Before you run any command, understand the three root causes of container failure: resource exhaustion (OOM, disk full), application bugs (segfaults, uncaught exceptions), and infrastructure issues (Docker daemon restart, network timeouts). Each leaves a different signature.

OOM kills leave exit code 137 and a kernel log. Disk full leaves exit code 1 with a 'no space left on device' error in logs. Segfaults leave exit code 139. Infrastructure issues often leave exit code 125 (daemon error) or 126 (command not executable).

Your first step is always: check the exit code. docker inspect <container> --format '{{.State.ExitCode}}'. Then match it to the list above. This narrows your search from everything to one category.

#!/bin/bash
# io.thecodeforge — DevOps tutorial
# Check exit code of a stopped container
docker inspect my_app --format '{{.State.ExitCode}}'
# Output: 137
# Then check OOM logs
dmesg | grep -i oom | tail -5
# Output: [12345.678] oom-kill: constraint=CONSTRAINT_MEMCG ...

Memory Troubleshooting: Beyond the Heap

Memory issues are the #1 cause of container crashes in production. But most devs only look at the app heap. That's like checking your car's fuel gauge when the engine seizes — you're missing the oil pressure.

Inside a container, memory is used by: the app heap, native code (JNI, C libraries), thread stacks, Metaspace (JVM), page cache, and tmpfs mounts. All of it counts against the memory limit. If the sum exceeds the limit, the kernel kills the container.

To see real memory usage: docker stats <container>. Look at the MEM USAGE / LIMIT column. If it's close to 100%, you're on the edge. Then use docker exec <container> cat /sys/fs/cgroup/memory/memory.usage_in_bytes to get the exact cgroup value.

For JVM apps, add -XX:NativeMemoryTracking=summary and use jcmd <pid> VM.native_memory to see non-heap usage. For Node.js, check process.memoryUsage().

#!/bin/bash
# io.thecodeforge — DevOps tutorial
# Check memory usage inside container
docker stats my_app --no-stream
# Output:
# CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %
# abc123         my_app    45.2%     3.8GiB / 4GiB       95%
# Check cgroup memory
docker exec my_app cat /sys/fs/cgroup/memory/memory.usage_in_bytes
# Output: 4080218931 (about 3.8GiB)
# Check OOM kills for this container
dmesg | grep -i 'my_app' | grep -i oom
# Output: (if any OOM kill happened)

Disk Full: The Silent Killer

A full disk doesn't crash your container — it freezes it. Writes fail silently, logs stop, and health checks time out. Then your orchestrator kills the container for being unhealthy. The real fix isn't restarting; it's freeing space.

First, check disk usage on the host: df -h. If any mount is at 100%, that's your problem. Docker stores images, containers, and volumes under /var/lib/docker by default. Use docker system df to see how much space each component uses.

Common culprits: unused images (docker image prune -a), stopped containers (docker container prune), and dangling volumes (docker volume prune). For production, set up a cron job to run docker system prune -f --volumes weekly.

But sometimes the issue is inside the container — logs growing unbounded. Always mount a log directory to a dedicated volume with a size limit, or use a log shipper that rotates.

#!/bin/bash
# io.thecodeforge — DevOps tutorial
# Check disk usage on host
df -h /var/lib/docker
# Output:
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/sda1        50G   48G  2.0G  96% /var/lib/docker
# Check Docker disk usage
docker system df
# Output:
# TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
# Images          10        5         20GB      10GB (50%)
# Containers      8         3         5GB       2GB (40%)
# Local Volumes   5         2         10GB      8GB (80%)
# Build Cache     0         0         0B        0B
# Prune unused resources
docker system prune -a --volumes -f

Zombie Processes and PID 1 Problems

Containers run with PID 1 — the init process. If your app doesn't handle signal forwarding and reaping zombie children, you get a mess. Zombie processes accumulate, PID 1 doesn't respond to SIGTERM, and your container becomes unkillable.

This is a classic rookie mistake: running a Java app directly as java -jar app.jar. Java doesn't reap zombies. If your app spawns child processes (e.g., forking a shell command), those children become zombies when they exit, because PID 1 (Java) doesn't call wait().

The fix: use a lightweight init system like tini or dumb-init. Add it to your Dockerfile: ENTRYPOINT ["/usr/bin/tini", "--"] then CMD ["java", "-jar", "app.jar"]. Tini handles signal forwarding and reaping.

To check for zombies inside a container: docker exec <container> ps aux | grep 'Z'. If you see zombie processes, you need an init system.

# io.thecodeforge — DevOps tutorial
FROM openjdk:11-jre-slim
# Install tini
RUN apt-get update && apt-get install -y tini && rm -rf /var/lib/apt/lists/*
# Use tini as init
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["java", "-jar", "app.jar"]

Network Timeouts and DNS Resolution

Your container can't reach the database. But it worked yesterday. Network issues in Docker are often DNS-related. By default, containers use the host's DNS via /etc/resolv.conf. If that file is stale or the DNS server is unreachable, name resolution fails.

First, test DNS inside the container: docker exec <container> nslookup db.example.com. If it fails, check the container's /etc/resolv.conf: docker exec <container> cat /etc/resolv.conf. It should list valid nameservers.

Common fix: set --dns flag to a reliable DNS server like 8.8.8.8. Or use --dns-search to add search domains. For production, run a local DNS cache like dnsmasq in a sidecar container.

Another gotcha: Docker's default bridge network doesn't allow DNS resolution by container name. Use a user-defined network: docker network create mynet then docker run --network mynet. This enables DNS resolution using container names as hostnames.

#!/bin/bash
# io.thecodeforge — DevOps tutorial
# Test DNS inside container
docker exec my_app nslookup db.example.com
# Output:
# Server:         8.8.8.8
# Address:        8.8.8.8#53
# Non-authoritative answer:
# Name:   db.example.com
# Address: 10.0.0.1
# Check resolv.conf
docker exec my_app cat /etc/resolv.conf
# Output:
# nameserver 8.8.8.8
# nameserver 8.8.4.4

Health Checks: The First Line of Defense

You shouldn't wait for a crash to know something's wrong. Health checks let Docker monitor your container and restart it automatically if it becomes unhealthy. But most people write health checks that are too simple — they check if the process is running, not if the app is actually working.

A good health check tests a real endpoint: curl -f http://localhost:8080/health. It should return HTTP 200 within a timeout. Set --health-interval=30s, --health-retries=3, and --health-start-period=30s to give the app time to start.

In your Dockerfile: HEALTHCHECK --interval=30s --timeout=3s --start-period=30s --retries=3 CMD curl -f http://localhost:8080/health || exit 1.

To see health status: docker inspect --format '{{.State.Health.Status}}' <container>. It returns healthy, unhealthy, or starting. If it's unhealthy, check the last 5 health check logs: docker inspect --format '{{json .State.Health.Log}}' <container>.

# io.thecodeforge — DevOps tutorial
FROM nginx:alpine
# Install curl for health check
RUN apk add --no-cache curl
# Add health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=30s --retries=3 \
  CMD curl -f http://localhost/ || exit 1
EXPOSE 80