Docker Healthchecks and Restart Policies: Stop Restarting Dead Containers

Your container is running. But is it working? I've seen a payment service that was 'up' for 3 days while silently dropping every transaction. The process was alive. The thread pool was dead. Docker had no idea. That's the lie of 'just check if the process is running'.

Healthchecks solve this by asking the container: 'Are you actually doing your job?' Restart policies then decide: 'Should I try again or give up?' Together they turn fragile containers into self-healing services. Without them, you're flying blind — your monitoring says 'green' while your users see errors.

By the end of this, you'll write healthchecks that catch real failures, configure restart policies that prevent infinite restart loops, and know exactly what to do when your container enters a crash loop at 3 AM.

Why Process Alive Doesn't Mean Service Healthy

Docker's default behavior is naive: if the PID 1 process is alive, the container is 'Up'. But your app can be in a death spiral — deadlocked threads, exhausted connection pools, corrupted state — while the process breathes. I've debugged a Kafka consumer that was 'Up' for 12 hours while consuming zero messages because a deserialization error had thrown an uncaught exception in a callback. The main thread was fine. The business logic was dead.

A healthcheck changes this. It's a command Docker runs periodically inside the container. Exit code 0 means healthy. Exit code 1 means unhealthy. Docker tracks the state and can act on it — restart the container, stop routing traffic, alert your team.

Without a healthcheck, you're trusting the OS process scheduler to tell you if your app works. It won't. Healthchecks are the difference between 'the container is running' and 'the service is serving'.

// io.thecodeforge — DevOps tutorial

# A Dockerfile with a basic HTTP healthcheck
FROM node:18-alpine

WORKDIR /app
COPY package*.json ./
RUN npm install --production
COPY . .

EXPOSE 3000

# Healthcheck: curl the /health endpoint every 30s, timeout after 10s
# Start checking after 5s (give app time to boot)
# Mark unhealthy after 3 consecutive failures
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1

CMD ["node", "server.js"]

Writing Healthchecks That Catch Real Failures

A healthcheck must test what matters. For a web service, hit an endpoint that exercises the database connection pool, not just a static 'OK' string. I've seen healthchecks that returned 200 while the database was down because the endpoint didn't touch the DB. That's a lying healthcheck.

For a worker process, check that it's processing messages. For a database, run a simple query. The healthcheck should be fast — under 5 seconds ideally. If it takes 30 seconds, you're adding load during a crisis.

Use the 'start-period' flag to give your app time to initialize. Without it, the healthcheck might fail during boot and trigger a restart loop. Set 'retries' to 3-5 to avoid flapping on transient issues.

// io.thecodeforge — DevOps tutorial

# Healthcheck that tests DB connectivity via a dedicated endpoint
FROM node:18-alpine

# Install curl for healthcheck
RUN apk add --no-cache curl

WORKDIR /app
COPY package*.json ./
RUN npm install --production
COPY . .

EXPOSE 3000

# Healthcheck hits /health/db which does a SELECT 1
# If DB is down, this returns 503 and healthcheck fails
HEALTHCHECK --interval=15s --timeout=5s --start-period=10s --retries=3 \
  CMD curl -f http://localhost:3000/health/db || exit 1

CMD ["node", "server.js"]

Restart Policies: When to Restart and When to Give Up

Restart policies tell Docker what to do when a container exits. The options: 'no' (never restart), 'on-failure' (restart if exit code non-zero), 'always' (always restart, even if you stop it manually), 'unless-stopped' (always restart unless you explicitly stop it).

'always' is dangerous — if your container crashes on startup due to a config error, it will restart forever, burning CPU and filling logs. 'on-failure' with a max retry count is safer. 'unless-stopped' is my default for most services — it survives daemon restarts but respects manual stops.

I once saw a team use 'always' on a container that had a typo in the command. It restarted 10,000 times in an hour, generating gigabytes of logs and costing $500 in CloudWatch bills. Set limits.

// io.thecodeforge — DevOps tutorial

version: '3.8'
services:
  api:
    build: .
    ports:
      - "3000:3000"
    # Restart unless explicitly stopped, max 5 retries on failure
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s

  worker:
    build: ./worker
    # Restart on failure, but only up to 3 times
    restart: on-failure:3
    depends_on:
      - api

Combining Healthchecks and Restart Policies for Self-Healing

Healthchecks and restart policies are a tag team. The healthcheck detects unhealthiness. The restart policy decides what to do. But they don't automatically restart on healthcheck failure — that's a common misconception. By default, Docker only restarts on container exit, not on unhealthy status.

To restart on unhealthy, you need an orchestrator like Docker Swarm or Kubernetes, or a custom watcher. In Swarm, set '--restart-condition=any' and the service will restart when healthcheck fails. In Kubernetes, liveness probes restart the pod.

For standalone Docker, you can use a sidecar script that watches 'docker inspect' and restarts the container. But honestly, if you need that, you should be using an orchestrator.

// io.thecodeforge — DevOps tutorial

version: '3.8'
services:
  api:
    image: myapp:latest
    deploy:
      replicas: 3
      restart_policy:
        condition: any  # Restart on exit OR unhealthy
        delay: 5s
        max_attempts: 3
        window: 120s
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 15s
      timeout: 5s
      retries: 2
      start_period: 10s

Gotchas from the Trenches: Healthcheck Pitfalls

1. Healthcheck command not found: If your image lacks curl, the healthcheck fails immediately. Always test your healthcheck command inside the container first.
2. Healthcheck too slow: A healthcheck that takes 30 seconds during a traffic spike can cause cascading failures. Keep it under 5 seconds.
3. Healthcheck port mismatch: If your app listens on port 8080 internally but you healthcheck port 3000, it fails. Use 'docker exec' to verify the port.
4. Start period too short: If your app takes 20 seconds to boot but start-period is 5s, the healthcheck fails during boot and may trigger unwanted restarts.
5. Restart policy 'always' with healthcheck: If the healthcheck fails but the process doesn't exit, 'always' won't restart. You need an orchestrator or a custom script.

// io.thecodeforge — DevOps tutorial

# Debug healthcheck: run the same command inside the container
docker exec <container> curl -f http://localhost:3000/health
# Check health status
docker inspect --format='{{json .State.Health}}' <container>
# View healthcheck logs
docker inspect --format='{{json .State.Health.Log}}' <container> | jq .

When Not to Use Healthchecks and Restart Policies

Healthchecks add complexity. For batch jobs that run once and exit, skip them. For containers that are part of a unit test suite, restart policies are pointless — you want them to exit cleanly.

If your app is stateless and crashes on any error, a simple 'on-failure:3' is enough. Don't over-engineer with healthchecks if the process itself is the failure detector.

For stateful services like databases, be careful with restart policies. A database that restarts on failure might come back with corrupted data. Use 'on-failure:2' and alert manually. Healthchecks for databases should be read-only queries that don't affect performance.

// io.thecodeforge — DevOps tutorial

# PostgreSQL healthcheck: lightweight query
FROM postgres:15

# Healthcheck runs a simple SELECT 1
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=5 \
  CMD pg_isready -U postgres || exit 1

# Or use a custom script that checks replication lag
# HEALTHCHECK --interval=10s CMD /healthcheck.sh