Docker Volumes — Why `-v` Deleted 3 Days of Postgres Data

#!/bin/bash
# ── STEP 1: Create a named volume Docker will manage ──────────────────────────
docker volume create postgres_data

# ── STEP 2: Inspect the volume to see where Docker actually stores it ─────────
docker volume inspect postgres_data
# Output shows the 'Mountpoint' on the host — usually /var/lib/docker/volumes/...

# ── STEP 3: Run Postgres and mount our volume to its data directory ───────────
# The -v flag maps: <volume_name>:<path_inside_container>
docker run -d \n  --name postgres_primary \n  -e POSTGRES_USER=appuser \n  -e POSTGRES_PASSWORD=securepass123 \n  -e POSTGRES_DB=appdb \n  -v postgres_data:/var/lib/postgresql/data \n  postgres:15-alpine

# ── STEP 4: Write some data into the database ─────────────────────────────────
docker exec -it postgres_primary psql -U appuser -d appdb \n  -c "CREATE TABLE users (id SERIAL PRIMARY KEY, name TEXT);"
docker exec -it postgres_primary psql -U appuser -d appdb \n  -c "INSERT INTO users (name) VALUES ('Alice'), ('Bob');"

# ── STEP 5: Destroy the container completely ──────────────────────────────────
# Without a volume this would delete all our data. Watch what happens instead.
docker rm -f postgres_primary

# ── STEP 6: Start a BRAND NEW container using the SAME volume ─────────────────
docker run -d \n  --name postgres_restored \n  -e POSTGRES_USER=appuser \n  -e POSTGRES_PASSWORD=securepass123 \n  -e POSTGRES_DB=appdb \n  -v postgres_data:/var/lib/postgresql/data \n  postgres:15-alpine

# ── STEP 7: Prove the data survived the container deletion ────────────────────
docker exec -it postgres_restored psql -U appuser -d appdb \n  -c "SELECT * FROM users;"

# ── BACKUP a named volume ─────────────────────────────────────────────────────
# Run a temporary container that mounts the volume and creates a tar archive
docker run --rm \n  -v postgres_data:/source:ro \n  -v $(pwd)/backups:/backup \n  alpine tar czf /backup/postgres_data_$(date +%Y%m%d).tar.gz -C /source .
# The :ro flag makes the volume read-only inside this container

# ── RESTORE a named volume from backup ────────────────────────────────────────
# Stop the container using the volume first
docker stop postgres_primary

# Extract the backup into the volume
docker run --rm \n  -v postgres_data:/target \n  -v $(pwd)/backups:/backup:ro \n  alpine tar xzf /backup/postgres_data_20260405.tar.gz -C /target

# Restart the container
docker start postgres_primary

# ── CLEANUP (optional — only run when you're done experimenting) ──────────────
# docker rm -f postgres_restored && docker volume rm postgres_data

#!/bin/bash
# ── STEP 1: Create a user-defined bridge network ──────────────────────────────
# This is what enables DNS-based container discovery
docker network create \n  --driver bridge \n  --subnet 172.28.0.0/16 \n  app_internal_network

# ── STEP 2: Start a Redis cache container on our custom network ───────────────
# Notice: we do NOT publish Redis's port to the host (no -p flag)
# Only containers on app_internal_network can reach it — that's intentional
docker run -d \n  --name redis_cache \n  --network app_internal_network \n  redis:7-alpine

# ── STEP 3: Start an API container on the same network ───────────────────────
# It can reach Redis using the hostname 'redis_cache' — not an IP address
docker run -d \n  --name api_server \n  --network app_internal_network \n  -p 3000:3000 \n  -e REDIS_HOST=redis_cache \n  -e REDIS_PORT=6379 \n  node:20-alpine \n  sh -c "npm install -g redis-cli && sleep infinity"

# ── STEP 4: Prove DNS resolution works inside the network ─────────────────────
# From inside api_server, ping redis_cache by NAME (not IP)
docker exec api_server ping -c 3 redis_cache

# ── STEP 5: Inspect the network to see both containers connected ──────────────
docker network inspect app_internal_network \n  --format '{{range .Containers}}{{.Name}}: {{.IPv4Address}}{{"\n"}}{{end}}'

# ── STEP 6: Verify Redis is NOT reachable from your host machine directly ─────
# This should fail — Redis is internal only, which is exactly what we want
curl -v telnet://localhost:6379 2>&1 | head -5

# ── Network isolation: create a second network for the frontend ───────────────
docker network create frontend_network

# Frontend container on frontend_network — CANNOT reach redis_cache
docker run -d \n  --name nginx_frontend \n  --network frontend_network \n  -p 80:80 \n  nginx:alpine

# Verify isolation: nginx_frontend cannot reach redis_cache
docker exec nginx_frontend ping -c 1 redis_cache 2>&1
# ping: bad address 'redis_cache' — DNS resolution fails across networks

# Connect nginx to BOTH networks to enable communication
docker network connect app_internal_network nginx_frontend
# Now nginx can resolve and reach redis_cache

# ── CLEANUP ───────────────────────────────────────────────────────────────────
# docker rm -f redis_cache api_server nginx_frontend
# docker network rm app_internal_network frontend_network

# io/thecodeforge/docker-compose.production.yml
# Runs a Node.js API backed by Postgres with persistent storage
# and a private network — no exposed database ports

version: "3.9"

services:

  # ── Postgres Database ─────────────────────────────────────────────────────
  database:
    image: postgres:15-alpine
    container_name: postgres_primary
    restart: unless-stopped          # Restart on crash, but respect manual stops
    environment:
      POSTGRES_USER: appuser
      POSTGRES_PASSWORD: securepass123
      POSTGRES_DB: appdb
    volumes:
      # Named volume keeps data safe across container replacements
      - postgres_data:/var/lib/postgresql/data
      # Seed script runs once on first volume initialization
      - ./db/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
    networks:
      - backend_network
    healthcheck:
      # pg_isready exits 0 when Postgres is accepting connections
      test: [\"CMD-SHELL\", \"pg_isready -U appuser -d appdb\"]\n      interval: 10s      # Check every 10 seconds\n      timeout: 5s        # Fail check if no response in 5s\n      retries: 5         # Mark unhealthy after 5 failures\n      start_period: 30s  # Grace period before checks start\n    # NO ports: section — database is intentionally unreachable from host\n\n  # ── Node.js REST API ──────────────────────────────────────────────────────\n  api:\n    build:\n      context: ./api\n      dockerfile: Dockerfile\n    container_name: node_api\n    restart: unless-stopped\n    ports:\n      - \"3000:3000\"      # Only the API is exposed to the outside world\n    environment:\n      # Uses the service name 'database' as the hostname — Docker DNS resolves it\n      DATABASE_URL: postgresql://appuser:securepass123@database:5432/appdb\n      NODE_ENV: production\n      PORT: 3000\n    volumes:\n      # Bind mount for uploaded files — you want these on the host for backups\n      - ./uploads:/app/uploads\n    networks:\n      - backend_network\n    depends_on:\n      database:\n        # 'service_healthy' waits for the healthcheck to pass — not just container start\n        condition: service_healthy\n\n# ── Named Volumes ─────────────────────────────────────────────────────────────\n# Declared here so Docker Compose manages their lifecycle\nvolumes:\n  postgres_data:\n    driver: local\n    # Optional: add 'external: true' if this volume is managed outside Compose\n\n# ── Networks ──────────────────────────────────────────────────────────────────\nnetworks:\n  backend_network:\n    driver: bridge\n    # ipam config is optional but useful when you need predictable subnets\n    ipam:\n      config:\n        - subnet: 172.29.0.0/16",
        "output": "$ docker compose -f io/thecodeforge/docker-compose.production.yml up -d\n\n[+] Running 4/4\n ✔ Network app_backend_network     Created    0.1s\n ✔ Volume  postgres_data           Created    0.1s\n ✔ Container postgres_primary      Healthy    18.3s\n ✔ Container node_api              Started    18.5s\n\n# Notice: postgres_primary reaches 'Healthy' BEFORE node_api starts\n# That's the healthcheck + depends_on working as intended\n\n$ docker compose ps\nNAME               IMAGE               STATUS\npostgres_primary   postgres:15-alpine  Up 25 seconds (healthy)\nnode_api           api-api             Up 7 seconds\n\n$ curl http://localhost:3000/health\n{\"status\":\"ok\",\"database\":\"connected\",\"uptime\":12.4}"
      },
      "callout": {
        "type": "mental_model",
        "title": "depends_on as a Bouncer at a Club",
        "text": "Think of depends_on as a bouncer at a club. Without condition: service_healthy, the bouncer only checks that the club door is open (container started) — they do not check if the DJ has set up (process ready). With condition: service_healthy, the bouncer actually walks inside and confirms the DJ is playing music (healthcheck passes) before letting the next guest in.",
        "hook": "What is the difference between depends_on with no condition and depends_on with condition: service_healthy?",
        "bullets": [
          "No condition: waits for the container to START (docker start returns). The process inside may still be initializing.",
          "service_healthy: waits for the healthcheck to PASS. The process inside must be fully ready and accepting connections.",
          "For databases, message queues, and any service with startup time, always use service_healthy.",
          "For services that start instantly (static file servers, simple APIs), no condition is sufficient."
        ]
      },
      "production_insight": "The start_period: 30s flag on the healthcheck is critical for services with slow startup. Without it, the healthcheck runs immediately and may fail before the service is ready, causing unnecessary container restarts. Set start_period to the expected maximum startup time plus a buffer. For Postgres, 30-60 seconds is typical. For JVM applications, 60-120 seconds may be needed.",
      "decision_tree": {
        "title": "Startup Ordering Strategy",
        "items": [
          {
            "condition": "Database dependency (API depends on Postgres/MySQL)",
            "result": "depends_on with condition: service_healthy + healthcheck using pg_isready/mysqladmin ping. Always."
          },
          {
            "condition": "Cache dependency (API depends on Redis)",
            "result": "depends_on with condition: service_healthy + healthcheck using redis-cli ping."
          },
          {
            "condition": "Independent services with no dependency",
            "result": "No depends_on needed. Let Docker start them in any order."
          },
          {
            "condition": "Complex startup with multiple dependencies",
            "result": "Use a wait-for-it.sh or dockerize script in the entrypoint that polls all dependencies before starting the application."
          }
        ]
      },
      "key_takeaway": "depends_on without condition: service_healthy only waits for container start — not process readiness. Always combine depends_on with a healthcheck for databases and services with startup time. The start_period flag prevents false healthcheck failures during initialization. This pattern is the foundation of every reliable Docker Compose deployment."
    },
    {
      "heading": "Comparing Bridge, Host, and None Networking Modes",
      "content": "Docker provides three built-in network drivers that serve fundamentally different purposes. Choosing the wrong one causes connectivity issues, performance bottlenecks, or security holes. Here's a direct comparison of `bridge`, `host`, and `none` — when to use each and what you give up.\n\n**Bridge (default) — isolated network with NAT:** Every container gets its own network namespace. Traffic flows through a virtual bridge (`docker0` by default). Containers can communicate with each other (on a user-defined bridge) via embedded DNS. Published ports (`-p`) create iptables rules to forward host traffic into the container. Isolation is preserved: unless you explicitly publish a port, nothing from outside the network can reach the container. Use bridge for 90% of workloads — it's the safe default that balances isolation and connectivity.\n\n**Host — container shares the host's network stack:** The container's network namespace is the same as the host's. No bridge, no NAT, no iptables overhead. The container can bind to any host port and access any host network interface. Performance is identical to running the process directly on the host. The trade-off is zero isolation — every container on host network competes for ports and can see all host traffic. Use `--network host` only when you need maximum performance (e.g., high-throughput packet processing, real-time financial services) and you trust the container not to interfere with other services.\n\n**None — complete network isolation:** The container has no network interfaces except loopback (`lo`). It cannot reach any other container or the outside world, and nothing can reach it. This is not a bug — it's the strongest security profile Docker offers. Use `--network none` for batch jobs that process local files, for security audit tools that should have no network access, or for containers that only need to communicate via shared volumes or Unix sockets.\n\n**Failure scenario — host network collision:** A team moved a Redis container from bridge to host network to reduce latency. They didn't realize another service on the host was already using port 6379. Redis failed to start because the port was taken. Even after freeing the port, the Redis container was now visible to all processes on the host, bypassing Docker's network security. Lesson: only use host mode on dedicated hosts where no other container or host service uses the same ports.",
      "code": {
        "language": "bash",
        "filename": "io/thecodeforge/network_mode_demo.sh",
        "code": "#!/bin/bash\n# ── 1. Bridge mode (default) — isolated with NAT ────────────────────────────\ndocker run -d --name web_bridge --network bridge -p 8080:80 nginx:alpine\n# Inside the container: ifconfig shows eth0 with 172.17.0.x IP\n# Host sees port 8080 mapped to container's port 80\n\n# ── 2. Host mode — shares host network stack ──────────────────────────────────\ndocker run -d --name web_host --network host nginx:alpine\n# Inside the container: ifconfig shows the host's interfaces (eth0, lo, etc.)\n# Port 80 now directly on host — no -p flag needed (it binds to host's port 80)\n\n# ── 3. None mode — zero network access ───────────────────────────────────────\ndocker run -it --name job_isolated --network none alpine sh\n# Inside the container: only lo interface exists\n# ping google.com fails: ping: bad address 'google.com'\n# exit\n\n# ── Verify differences ───────────────────────────────────────────────────────\ndocker exec web_bridge hostname -I           # Shows container's private IP\ndocker exec web_host hostname -I             # Shows host's IP (same as host)\ndocker exec job_isolated hostname -I         # Shows nothing (127.0.0.1 only)\n\n# ── Cleanup ───────────────────────────────────────────────────────────────────\ndocker rm -f web_bridge web_host job_isolated",
        "output": "$ docker exec web_bridge hostname -I\n172.17.0.2\n\n$ docker exec web_host hostname -I\n192.168.1.100\n\n$ docker exec job_isolated hostname -I\n\n# (empty output — container has no external IP)"
      },
      "callout": {
        "type": "warning",
        "title": "Host Mode Security Risk",
        "text": "Never use --network host on a shared host or in a multi-tenant environment. The container can listen on any port, access any interface, and potentially interfere with other containers and host processes. Only use host mode on dedicated single-purpose servers after auditing the host's port usage."
      },
      "production_insight": "The host network mode bypasses Docker's NAT overhead, which can reduce latency by 1-3ms per packet. For most applications this difference is negligible. Only invest in host mode if your profiling shows network overhead is a bottleneck. Even then, consider using --network host with a read-only root filesystem and no cap-add to minimize the security blast radius.",
      "key_takeaway": "Bridge (default) provides network isolation with NAT. Host mode gives maximum performance at the cost of isolation. None mode completely air-gaps the container. Choose bridge for general workloads, host for performance-critical services on dedicated hosts, and none for security-sensitive batch jobs."
    },
    {
      "heading": "Internal DNS and Service Discovery in Docker",
      "content": "Docker's embedded DNS is the glue that makes service discovery work in user-defined networks. Without it, you'd have to hard-code IP addresses that change every time a container restarts. Here's how Docker DNS works under the hood and how to debug it when it breaks.\n\n**How Docker DNS works:** When you create a user-defined network (e.g., `docker network create mynet`), Docker starts an internal DNS resolver on the network's gateway IP (typically the bridge IP, like 172.18.0.1). Each container connected to that network registers its name (the `--name` or container name) as an A record. The resolver runs inside the Docker engine, not inside any container. Containers are configured to use this resolver via `/etc/resolv.conf` inside the container, which points to 127.0.0.11 — a small DNS proxy in each container that forwards queries to the engine's resolver.\n\n**Resolution scope:** DNS resolution only works for containers on the same user-defined network. Containers on different networks cannot resolve each other by name unless they are connected to both networks. The default bridge network does NOT have embedded DNS — this is a frequent source of confusion.\n\n**Custom DNS entries:** Docker DNS also resolves service names from Compose (the `service` name) and aliases specified with `--network-alias`. For example, `docker run --network mynet --network-alias db-primary --name postgres_1 ...` allows other containers to reach it as `db-primary`.\n\n**Debugging DNS:** If a container cannot resolve a peer's hostname, first verify they are on the same network: `docker network inspect <network>`. Then test DNS inside the container: `docker exec <container> nslookup <target>`. If nslookup fails with 'server can't find', the peer isn't registered. If nslookup succeeds but ping fails, the issue is a firewall or port binding (the peer exists but isn't listening).\n\n**Failure scenario — DNS proxy misconfiguration:** A team ran a custom DNS server inside a container that listened on port 53. When they tried to start the container, Docker's internal DNS proxy (127.0.0.11) could not start because port 53 was already taken inside the container network namespace. The container started but DNS resolution for other containers failed. Fix: avoid binding to port 53 inside containers; use a different port and remap in the application.",
      "code": {
        "language": "bash",
        "filename": "io/thecodeforge/dns_debug_demo.sh",
        "code": "#!/bin/bash\n# ── Set up a test network ───────────────────────────────────────────────────\ndocker network create dns_demo\n\n# ── Start a Redis container ──────────────────────────────────────────────────\ndocker run -d --name redis-cache --network dns_demo redis:7-alpine\n\n# ── Start an Alpine container for testing ────────────────────────────────────\ndocker run -it --name tester --network dns_demo alpine sh -c \"\n  # Check internal resolver config\n  cat /etc/resolv.conf\n  echo '---'\n  # Query DNS for redis-cache\n  nslookup redis-cache 2>&1 || echo 'nslookup not available'\n  # Try ping (built-in busybox)\n  ping -c 2 redis-cache\n  # Check if Docker DNS proxy is running\n  ss -tlnp | grep 53 || echo 'No listener on port 53 inside container'\n\"\n\n# ── Inspect network to see registered names ──────────────────────────────────\ndocker network inspect dns_demo --format '{{range .Containers}}{{.Name}}: {{.IPv4Address}}{{\"\\n\"}}{{end}}'\n\n# ── Test resolution from outside the network (should fail) ───────────────────\ndocker run --rm alpine nslookup redis-cache 2>&1 || echo 'DNS resolution fails from default bridge'\n\n# ── Cleanup ──────────────────────────────────────────────────────────────────\ndocker rm -f redis-cache tester\ndocker network rm dns_demo",
        "output": "$ docker exec tester cat /etc/resolv.conf\nnameserver 127.0.0.11\noptions ndots:0\n\n$ docker exec tester nslookup redis-cache\nName:      redis-cache\nAddress 1: 172.19.0.2 redis-cache.dns_demo\n\n$ docker exec tester ping -c 2 redis-cache\nPING redis-cache (172.19.0.2): 56 data bytes\n64 bytes from 172.19.0.2: seq=0 ttl=64 time=0.1 ms\n64 bytes from 172.19.0.2: seq=1 ttl=64 time=0.1 ms\n\n$ docker network inspect dns_demo --format '...'\nredis-cache: 172.19.0.2/16\ntester: 172.19.0.3/16"
      },
      "callout": {
        "type": "info",
        "title": "DNS Resolution Only on User-Defined Networks",
        "text": "The default bridge network does not support DNS-based service discovery. If you see 'could not resolve hostname' errors, the most likely cause is that your containers are on the default bridge. Create a user-defined network with `docker network create` and attach both containers."
      },
      "production_insight": "In production Swarm or Kubernetes clusters, service discovery is handled differently (by the orchestrator's DNS system). In Docker Compose, the embedded DNS is sufficient. For multi-host deployments, use an overlay network with `--opt encrypted` to secure DNS traffic between nodes. The default Docker DNS resolver has a TTL of 600 seconds — if you need faster propagation of network changes, adjust the `--dns-opt` flags or use a third-party solution like Consul DNS.",
      "key_takeaway": "Docker's embedded DNS on user-defined networks resolves container names automatically. The resolver runs inside the Docker engine, not in containers. DNS only works within the same user-defined network. Debug resolution with `nslookup` inside the container. Never rely on default bridge DNS — it doesn't exist."
    },
    {
      "heading": "Volume Types Compared: Named Volume vs Bind Mount vs tmpfs",
      "content": "Docker offers three distinct storage options. Each behaves differently in terms of lifecycle, performance, and security. Choosing the right one depends on whether you need persistence, speed, or isolation from the host filesystem.\n\n**Named Volume** — Managed by Docker, stored under `/var/lib/docker/volumes/`. Docker handles creation, deletion, and permissions. This is the production default for stateful services like databases. The volume can be backed up using `docker volume inspect` and restored via helper containers. Named volumes are portable across hosts when using remote volume drivers.\n\n**Bind Mount** — Maps an exact host path into the container. You control the host-side location and permissions. Bind mounts bypass Docker's volume management — they are direct filesystem mounts. This is ideal for development (live code reload, config injection) but introduces host filesystem dependency and permission issues. A bind mount is also the only way to mount a directory from the host that exists outside Docker's control.\n\n**tmpfs Mount** — Stored in host memory only. Never written to disk. Data is lost when the container stops. This is perfect for secrets, session tokens, or any data that must never be persisted. Performance is extremely fast (RAM speed), but limited by available memory. Use `--tmpfs /app/secrets:size=100m` to control the maximum size.\n\n**Decision table:**
| Property | Named Volume | Bind Mount | tmpfs |
|---|---|---|---|
| Management | Docker | User | Docker |
| Persistence | Yes (until `docker volume rm`) | Yes (host file) | No (lost on container stop) |
| Host path visibility | Hidden (under /var/lib/docker) | Explicit (you set the path) | N/A (memory) |
| Backup method | Container + tar | Standard filesystem tools | Not applicable |
| Performance (Linux) | Native | Native | RAM-speed |
| Security | Good (Docker manages permissions) | Caution (depends on host permissions) | Excellent (no disk) |
| Use case | Production databases, stateful services | Development code, config files | Secrets, session data |\n\n**Failure scenario — tmpfs size exceeded:** A team used tmpfs for a session store without setting a `size` limit. The application created large session objects (base64-encoded binary data). The tmpfs filled up and the container started crashing with 'no space left on device'. The host had plenty of free RAM, but the tmpfs mount was limited to 64MB by default. Fix: set `--tmpfs /session:size=500m` to give the session store adequate room.",
      "callout": {
        "type": "mental_model",
        "title": "Three Types of Storage, Three Mindsets",
        "text": "Named volumes are like a managed storage locker (safe, abstracted). Bind mounts are like a desk drawer on your host (convenient but you control the organization). tmpfs is like a whiteboard (fast, temporary, erased when you leave the room). Never put durable data on tmpfs, and never put secrets in bind mounts that might leak to disk.",
        "hook": "What happens when you mount a named volume over a directory that already has data from the image?",
        "bullets": [
          "If the volume is empty (first use), Docker copies the image's content into the volume before mounting.",
          "If the volume already has data (from a previous mount), the image's content is hidden under the mount — the container sees only the volume's data.",
          "This is why you can reattach a volume after a container rebuild and still see old data, but init scripts that expect an empty directory may not re-run."
        ]
      },
      "production_insight": "For production databases, always use named volumes with the local driver unless you have specific multi-host requirements. Bind mounts should be avoided in production due to permission complexity and lack of portability. tmpfs is excellent for temporary caches and secrets, but monitor memory usage — a rogue process can fill tmpfs and cause OOM-like failures. Set explicit size limits on all tmpfs mounts.",
      "key_takeaway": "Named volumes are for production persistence (managed by Docker), bind mounts are for development (direct host access), and tmpfs is for ephemeral in-memory data (secrets, caches). Choose based on lifecycle requirements: do you need data to survive container removal, or is it okay to lose it?"
    }
  ]