Docker Healthchecks and Restart Policies: Stop Restarting Dead Containers

Your container is running. But is it working? I've seen a payment service that was 'up' for 3 days while silently dropping every transaction. The process was alive. The thread pool was dead. Docker had no idea. That's the lie of 'just check if the process is running'.

Healthchecks solve this by asking the container: 'Are you actually doing your job?' Restart policies then decide: 'Should I try again or give up?' Together they turn fragile containers into self-healing services. Without them, you're flying blind — your monitoring says 'green' while your users see errors.

By the end of this, you'll write healthchecks that catch real failures, configure restart policies that prevent infinite restart loops, and know exactly what to do when your container enters a crash loop at 3 AM.

Why Process Alive Doesn't Mean Service Healthy

Docker's default behavior is naive: if the PID 1 process is alive, the container is 'Up'. But your app can be in a death spiral — deadlocked threads, exhausted connection pools, corrupted state — while the process breathes. I've debugged a Kafka consumer that was 'Up' for 12 hours while consuming zero messages because a deserialization error had thrown an uncaught exception in a callback. The main thread was fine. The business logic was dead.

A healthcheck changes this. It's a command Docker runs periodically inside the container. Exit code 0 means healthy. Exit code 1 means unhealthy. Docker tracks the state and can act on it — restart the container, stop routing traffic, alert your team.

Without a healthcheck, you're trusting the OS process scheduler to tell you if your app works. It won't. Healthchecks are the difference between 'the container is running' and 'the service is serving'.

// io.thecodeforge — DevOps tutorial

# A Dockerfile with a basic HTTP healthcheck
FROM node:18-alpine

WORKDIR /app
COPY package*.json ./
RUN npm install --production
COPY . .

EXPOSE 3000

# Healthcheck: curl the /health endpoint every 30s, timeout after 10s
# Start checking after 5s (give app time to boot)
# Mark unhealthy after 3 consecutive failures
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1

CMD ["node", "server.js"]

Writing Healthchecks That Catch Real Failures

A healthcheck must test what matters. For a web service, hit an endpoint that exercises the database connection pool, not just a static 'OK' string. I've seen healthchecks that returned 200 while the database was down because the endpoint didn't touch the DB. That's a lying healthcheck.

For a worker process, check that it's processing messages. For a database, run a simple query. The healthcheck should be fast — under 5 seconds ideally. If it takes 30 seconds, you're adding load during a crisis.

Use the 'start-period' flag to give your app time to initialize. Without it, the healthcheck might fail during boot and trigger a restart loop. Set 'retries' to 3-5 to avoid flapping on transient issues.

// io.thecodeforge — DevOps tutorial

# Healthcheck that tests DB connectivity via a dedicated endpoint
FROM node:18-alpine

# Install curl for healthcheck
RUN apk add --no-cache curl

WORKDIR /app
COPY package*.json ./
RUN npm install --production
COPY . .

EXPOSE 3000

# Healthcheck hits /health/db which does a SELECT 1
# If DB is down, this returns 503 and healthcheck fails
HEALTHCHECK --interval=15s --timeout=5s --start-period=10s --retries=3 \
  CMD curl -f http://localhost:3000/health/db || exit 1

CMD ["node", "server.js"]

Restart Policies: When to Restart and When to Give Up

Restart policies tell Docker what to do when a container exits. The options: 'no' (never restart), 'on-failure' (restart if exit code non-zero), 'always' (always restart, even if you stop it manually), 'unless-stopped' (always restart unless you explicitly stop it).

'always' is dangerous — if your container crashes on startup due to a config error, it will restart forever, burning CPU and filling logs. 'on-failure' with a max retry count is safer. 'unless-stopped' is my default for most services — it survives daemon restarts but respects manual stops.

I once saw a team use 'always' on a container that had a typo in the command. It restarted 10,000 times in an hour, generating gigabytes of logs and costing $500 in CloudWatch bills. Set limits.

// io.thecodeforge — DevOps tutorial

version: '3.8'
services:
  api:
    build: .
    ports:
      - "3000:3000"
    # Restart unless explicitly stopped, max 5 retries on failure
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s

  worker:
    build: ./worker
    # Restart on failure, but only up to 3 times
    restart: on-failure:3
    depends_on:
      - api

Combining Healthchecks and Restart Policies for Self-Healing

Healthchecks and restart policies are a tag team. The healthcheck detects unhealthiness. The restart policy decides what to do. But they don't automatically restart on healthcheck failure — that's a common misconception. By default, Docker only restarts on container exit, not on unhealthy status.

To restart on unhealthy, you need an orchestrator like Docker Swarm or Kubernetes, or a custom watcher. In Swarm, set '--restart-condition=any' and the service will restart when healthcheck fails. In Kubernetes, liveness probes restart the pod.

For standalone Docker, you can use a sidecar script that watches 'docker inspect' and restarts the container. But honestly, if you need that, you should be using an orchestrator.

// io.thecodeforge — DevOps tutorial

version: '3.8'
services:
  api:
    image: myapp:latest
    deploy:
      replicas: 3
      restart_policy:
        condition: any  # Restart on exit OR unhealthy
        delay: 5s
        max_attempts: 3
        window: 120s
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 15s
      timeout: 5s
      retries: 2
      start_period: 10s

Gotchas from the Trenches: Healthcheck Pitfalls

1. Healthcheck command not found: If your image lacks curl, the healthcheck fails immediately. Always test your healthcheck command inside the container first.
2. Healthcheck too slow: A healthcheck that takes 30 seconds during a traffic spike can cause cascading failures. Keep it under 5 seconds.
3. Healthcheck port mismatch: If your app listens on port 8080 internally but you healthcheck port 3000, it fails. Use 'docker exec' to verify the port.
4. Start period too short: If your app takes 20 seconds to boot but start-period is 5s, the healthcheck fails during boot and may trigger unwanted restarts.
5. Restart policy 'always' with healthcheck: If the healthcheck fails but the process doesn't exit, 'always' won't restart. You need an orchestrator or a custom script.

// io.thecodeforge — DevOps tutorial

# Debug healthcheck: run the same command inside the container
docker exec <container> curl -f http://localhost:3000/health
# Check health status
docker inspect --format='{{json .State.Health}}' <container>
# View healthcheck logs
docker inspect --format='{{json .State.Health.Log}}' <container> | jq .

When Not to Use Healthchecks and Restart Policies

Healthchecks add complexity. For batch jobs that run once and exit, skip them. For containers that are part of a unit test suite, restart policies are pointless — you want them to exit cleanly.

If your app is stateless and crashes on any error, a simple 'on-failure:3' is enough. Don't over-engineer with healthchecks if the process itself is the failure detector.

For stateful services like databases, be careful with restart policies. A database that restarts on failure might come back with corrupted data. Use 'on-failure:2' and alert manually. Healthchecks for databases should be read-only queries that don't affect performance.

// io.thecodeforge — DevOps tutorial

# PostgreSQL healthcheck: lightweight query
FROM postgres:15

# Healthcheck runs a simple SELECT 1
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=5 \
  CMD pg_isready -U postgres || exit 1

# Or use a custom script that checks replication lag
# HEALTHCHECK --interval=10s CMD /healthcheck.sh

start_interval Parameter: Fine-Tuning Startup Health

Docker Compose v3.9+ introduced the start_interval parameter, which controls how frequently the healthcheck runs during the start_period. Before this, the healthcheck always ran at the normal interval even during startup — meaning a slow-booting app with a 30s interval might only get 1-2 attempts during the start_period before being marked unhealthy. With start_interval, you can poll more aggressively during boot (e.g., every 2 seconds) to catch the moment the service becomes ready. This is critical for services with unpredictable startup times. Additionally, Compose v2.20+ introduced a stricter requirement: if you use healthcheck with depends_on condition: service_healthy, the healthcheck must exist. Without it, Compose will error. Always pair healthchecks with condition: service_healthy for strict startup ordering.

// io.thecodeforge — DevOps tutorial

version: '3.9'
services:
  api:
    image: myapp:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 60s
      start_interval: 2s   # Poll every 2s during start_period
    ports:
      - "3000:3000"

  nginx:
    image: nginx:alpine
    depends_on:
      api:
        condition: service_healthy  # Requires Compose v2.20+

Database-Specific Healthcheck Commands

Each database has its own lightweight healthcheck tool. For PostgreSQL, pg_isready checks if the server is accepting connections with minimal overhead. For MySQL, mysqladmin ping does the same. For Redis, redis-cli ping returns PONG. For InfluxDB, use curl against the /health endpoint. The key is using the database-native tool rather than a generic curl command that might add load. When embedding these in Docker Compose, be aware of shell variable escaping: $$VAR (double dollar) prevents Docker Compose from interpolating the variable, passing it literally to the container. For example, test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER"] passes $POSTGRES_USER to the shell inside the container, letting it read the container's environment variable rather than Compose's. This pattern is essential for database healthchecks where credentials come from environment variables.

// io.thecodeforge — DevOps tutorial

version: '3.9'
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: secret
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER"]
      interval: 10s
      timeout: 3s
      retries: 5
      start_period: 15s

  mysql:
    image: mysql:8
    environment:
      MYSQL_ROOT_PASSWORD: rootpass
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      interval: 10s
      timeout: 3s
      retries: 5

  redis:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 3

  influxdb:
    image: influxdb:2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8086/health"]
      interval: 15s
      timeout: 5s
      retries: 3

Healthcheck Failure Debugging: Common Scenarios

When healthchecks fail, the container gets stuck in 'starting' state after the start_period expires. Three common root causes stand out. First, the healthcheck command doesn't exist in the image — especially curl or wget on Alpine-based images. Alpine uses BusyBox and ships neither. Always RUN apk add --no-cache curl if your healthcheck depends on it. Second, cascading startup failures: when service A depends_on service B (condition: service_healthy), and B is slow to initialize, A's healthcheck may fail before B is ready, creating a deadlock-like scenario. Solve this with generous start_period values and start_interval for aggressive polling. Third, the healthcheck endpoint itself is working but the service behind it isn't — a healthcheck that returns 200 but doesn't exercise the actual service logic (database connection, message queue, external API). This is the 'lying healthcheck' pattern and is the most dangerous because everything looks green while the service silently fails.

// io.thecodeforge — DevOps tutorial

# 1. Check if command exists inside container
docker exec <container> which curl  # returns path or "not found"
docker exec <container> curl --version

# 2. Manually run the healthcheck command
docker exec <container> curl -f http://localhost:3000/health

# 3. Check health status and log
docker inspect --format='{{json .State.Health}}' <container> | jq .

# 4. View the last 5 healthcheck outputs
docker inspect --format='{{json .State.Health.Log}}' <container> | jq '.[-5:]'

# 5. Check dependency startup order
docker compose logs --tail=50 | grep -E "(healthy|unhealthy|starting)"

# 6. For Alpine images, install missing tools
# RUN apk add --no-cache curl  # Add to Dockerfile

Exit Code Reference for Healthchecks

Docker healthchecks follow a strict exit code convention that determines container health status. Exit code 0 means the container is healthy — the command succeeded and the service is working. Exit code 1 means the container is unhealthy — the command failed and the service is not working correctly. Exit code 2 is reserved by Docker and should not be used; if your healthcheck script returns 2, Docker treats it as an error state (effectively unhealthy). This simple convention is powerful: any command that returns 0 for success and non-zero for failure can serve as a healthcheck. Common examples: curl -f returns 0 on HTTP 200, non-zero otherwise; pg_isready returns 0 when PostgreSQL is accepting connections; redis-cli ping returns PONG with exit 0. You can chain commands with || exit 1 to coerce non-standard exit codes. A production pattern: healthcheck scripts that return exit code 0 only when all critical dependencies are available — database, cache, upstream API — and exit 1 otherwise.

// io.thecodeforge — DevOps tutorial

# Exit code 0: Healthy
docker exec <container> curl -f http://localhost:3000/health
echo $?  # 0

# Exit code 1: Unhealthy
docker exec <container> curl -f http://localhost:3000/down
echo $?  # 1

# Custom healthcheck script that checks multiple dependencies
cat > /healthcheck.sh << 'EOF'
#!/bin/sh
pg_isready -U postgres || exit 1
curl -f http://localhost:8080/internal/health || exit 1
redis-cli ping || exit 1
exit 0
EOF
chmod +x /healthcheck.sh

# In Dockerfile:
# HEALTHCHECK --interval=30s --timeout=10s --start-period=15s --retries=3 \
#   CMD /healthcheck.sh