Docker Monitoring and Logging: Stop Flying Blind in Production

You've deployed your microservices on Docker. Everything's running. Then at 3 AM, the pager goes off: 'Payment service latency > 10s.' You SSH in, run docker stats, and see one container eating 4GB of RAM. But which container? What was it doing? You have no logs, no metrics, no history. You're blind. That's the problem Docker monitoring and logging solves. By the end of this article, you'll have a production-ready stack that tells you exactly what's happening inside every container, past and present, so you can fix issues before they become incidents.

Why Default Docker Logging Will Bite You

Docker's default logging driver is json-file. It writes logs to the container's writable layer as JSON files. Sounds fine until you realize those logs live inside the container's storage, not on the host. If the container dies, logs die with it. Worse, no rotation means a chatty container can fill your disk. I've seen a single container write 20GB of logs in an hour, taking down the entire host. The fix: use a logging driver that ships logs off the container. journald for systemd integration, syslog for centralized logging, or gelf for Graylog. But the gold standard for modern stacks is the local driver: it's fast, uses binary format, and supports compression. Or use fluentd/loki for cloud-native setups. Never rely on docker logs in production — it's for debugging, not monitoring.

// io.thecodeforge — DevOps tutorial

// Run a container with local logging driver and rotation
docker run --log-driver local --log-opt max-size=10m --log-opt max-file=3 --name payment-service nginx

// Check current logging driver
docker info --format '{{.LoggingDriver}}'

// View logs from local driver
docker logs --tail 50 payment-service

Container Metrics: What to Collect and Why

Docker exposes raw container metrics via the Docker API and docker stats. But docker stats is a snapshot, not a history. For real monitoring, you need time-series data. The key metrics: CPU usage (as percentage of host), memory usage (RSS + cache), network I/O (bytes in/out), block I/O (read/write), and PIDs. Why PIDs? A fork bomb in a container can exhaust the host's PID limit. I've seen a misconfigured Node.js event loop spawn thousands of child processes, freezing the host. Collect these metrics every 15s using cAdvisor, which runs as a container and exposes Prometheus endpoints. Then ship to Prometheus. Don't forget to set resource limits — without them, one container can starve others.

// io.thecodeforge — DevOps tutorial

// Run cAdvisor container
docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:ro --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --volume=/dev/disk/:/dev/disk:ro --publish=8080:8080 --detach=true --name=cadvisor gcr.io/cadvisor/cadvisor:latest

// Verify metrics endpoint
curl http://localhost:8080/metrics | head -20

Prometheus: The Backbone of Container Monitoring

Prometheus scrapes metrics from cAdvisor and other exporters at configurable intervals. It's pull-based, which means you don't need to install agents — just expose an endpoint. For Docker monitoring, you'll scrape cAdvisor (container metrics), node_exporter (host metrics), and your application's custom metrics. The critical configuration: scrape interval (15s default), retention (15d default), and storage (local TSDB). In production, set retention to 30 days and use persistent volumes. Don't forget alerting rules — Prometheus can evaluate rules and fire alerts to Alertmanager. I once saw a team miss a memory leak because they only alerted on CPU. Alert on memory usage > 80% and container restarts.

// io.thecodeforge — DevOps tutorial

// prometheus.yml
scrape_configs:
  - job_name: 'cadvisor'
    scrape_interval: 15s
    static_configs:
      - targets: ['cadvisor:8080']
  - job_name: 'node'
    scrape_interval: 15s
    static_configs:
      - targets: ['node-exporter:9100']

// Run Prometheus with config
docker run --volume=/path/to/prometheus.yml:/etc/prometheus/prometheus.yml --publish=9090:9090 prom/prometheus

Grafana: Turning Metrics into Answers

Raw metrics are useless without visualization. Grafana connects to Prometheus and lets you build dashboards. For Docker monitoring, create a dashboard with: CPU usage per container (line chart), memory usage (bar chart), network I/O (area chart), and container restarts (stat panel). Use variables to filter by container name or host. The key: set up alerts in Grafana too — it can send notifications to Slack, PagerDuty, or email. Don't build one massive dashboard. Create separate dashboards for 'Container Overview', 'Host Health', and 'Application Performance'. I've seen teams spend weeks perfecting a dashboard that nobody uses. Start with the top 5 metrics that would have caught your last outage.

// io.thecodeforge — DevOps tutorial

// Run Grafana
docker run --publish=3000:3000 --volume=grafana-storage:/var/lib/grafana grafana/grafana

// Add Prometheus data source via API
curl -X POST http://admin:admin@localhost:3000/api/datasources -H 'Content-Type: application/json' -d '{"name":"Prometheus","type":"prometheus","url":"http://prometheus:9090","access":"proxy"}'

Log Aggregation with Loki: Logs as Cheap as Metrics

Loki is Grafana's log aggregation system designed for Kubernetes and Docker. Unlike Elasticsearch, Loki doesn't index the log content — it indexes only metadata (labels like container name, host). This makes it cheaper and faster for high-volume logs. Use Promtail (or Docker plugin) to ship logs to Loki. The Docker plugin is easiest: docker plugin install grafana/loki-docker-driver:latest --alias loki. Then set --log-driver=loki on containers. Query logs in Grafana with LogQL. Example: {container_name="payment-service"} |= "error". Loki is great for 'needle in a haystack' searches. But if you need full-text search on historical logs, Elasticsearch might be better. Trade-off: cost vs capability.

// io.thecodeforge — DevOps tutorial

// Install Loki Docker driver plugin
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

// Run Loki server
docker run --publish=3100:3100 --volume=loki-data:/loki grafana/loki:latest -config.file=/etc/loki/local-config.yaml

// Run container with Loki driver
docker run --log-driver=loki --log-opt loki-url=http://localhost:3100/loki/api/v1/push --log-opt max-size=10m --name webapp nginx

// Query logs via curl
curl -G -s 'http://localhost:3100/loki/api/v1/query_range' --data-urlencode 'query={container_name="webapp"}' | jq .

Structured Logging: The Missing Piece

Unstructured logs are a nightmare to query. 'Payment failed at 3:02' tells you nothing. Structured logging outputs JSON with fields like timestamp, level, service, trace_id, user_id, error. This lets you filter and aggregate in Loki. In Node.js, use pino or winston. In Python, structlog. In Go, logrus or zap. The key: include a correlation ID (trace ID) that spans services. Without it, you can't trace a request across microservices. I've spent hours correlating logs manually because a team didn't include trace IDs. Use OpenTelemetry for distributed tracing — it's the industry standard now.

// io.thecodeforge — DevOps tutorial

const pino = require('pino');
const logger = pino({
  level: 'info',
  formatters: {
    level(label) { return { level: label }; }
  },
  timestamp: pino.stdTimeFunctions.isoTime
});

// In request handler
app.get('/pay', (req, res) => {
  const traceId = req.headers['x-trace-id'] || crypto.randomUUID();
  logger.info({ traceId, userId: req.user.id, amount: req.body.amount }, 'Payment initiated');
  // ... process payment
  logger.error({ traceId, error: err.message }, 'Payment failed');
});

Alerting: Don't Wake Up for Everything

Alert fatigue is real. If every container restart triggers a page, you'll ignore the real ones. Define alerting rules in Prometheus or Grafana. Good alerts: 'Container memory > 85% for 5 minutes', 'Container restart count > 3 in 10 minutes', 'No logs from service for 5 minutes'. Bad alerts: 'CPU > 50%' (too noisy). Use severity levels: critical (page), warning (email), info (dashboard). I once worked at a company where every CPU spike paged the on-call. After a week, nobody responded. We fixed it by adding 'for' duration and only paging on sustained high usage. Also, set up a 'dead man's switch' — alert if Prometheus stops scraping.

// io.thecodeforge — DevOps tutorial

// alert.rules.yml
groups:
  - name: container_alerts
    rules:
      - alert: HighMemoryUsage
        expr: (container_memory_usage_bytes{container_label_com_docker_compose_service!=""} / container_spec_memory_limit_bytes{container_label_com_docker_compose_service!=""}) > 0.85
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Container {{ $labels.container_label_com_docker_compose_service }} memory > 85%"
      - alert: ContainerRestarts
        expr: rate(container_last_seen{container_label_com_docker_compose_service!=""}[5m]) > 0.3
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.container_label_com_docker_compose_service }} restarting frequently"

When Not to Use This Stack

Prometheus + Grafana + Loki is overkill for a single container on a dev machine. For local development, docker stats and docker logs are fine. For a small app with 2-3 containers, consider a lightweight alternative like Netdata (all-in-one monitoring) or Dozzle (log viewer). Also, if you're on Kubernetes, use the native monitoring stack (kube-prometheus-stack). Don't reinvent the wheel. And if you need full-text search on logs (e.g., for compliance audits), Elasticsearch + Kibana (ELK) might be better than Loki, despite the cost. Choose based on scale and requirements, not hype.

// io.thecodeforge — DevOps tutorial

// Run Netdata for quick monitoring
docker run --pid=host --privileged --volume=/:/host:ro,rslave --publish=19999:19999 netdata/netdata

// Run Dozzle for log viewing
docker run --volume=/var/run/docker.sock:/var/run/docker.sock --publish=8080:8080 amir20/dozzle