Airflow Docker Compose: The Setup That Wiped Our DAGs

The cleanup command looked harmless. One docker compose down -v to reclaim disk, one fresh up, and the entire Airflow history was gone.

DAGs, run history, connections, all of it lived in volumes that -v just deleted. The stack came up smiling and empty.

Compose is the fastest way to run Airflow and the fastest way to wipe it. You'll learn the volume and startup discipline here.

The Official Compose File, Service by Service

The official compose file is a small fleet: postgres for metadata, redis as Celery broker, scheduler, airflow-dag-processor, api-server on port 8080, workers, triggerer for deferrable tasks, and a one-shot airflow-init service, plus optional flower on port 5555 via --profile flower. Each container gets its role from the same image with a different command, which keeps versions consistent.

Fetch the versioned file with curl -LfO 'https://airflow.apache.org/docs/apache-airflow/3.3.1/docker-compose.yaml'. Read it service by service before changing anything. Postgres holds all state, redis holds task messages, scheduler orders work, workers execute, api-server renders the UI, triggerer runs deferrable event loops. Remove any piece without understanding its contract and the stack fails in confusing ways.

Pin the image tag to a tested Airflow version. Floating latest turns every recreate into a potential schema migration you didn't plan. Budget 4GB RAM minimum for Docker (8GB breathes), and use Compose v2.14+ since v1 can't parse the file. The compose quick-start carries no production security guarantees; graduate to the Helm chart when you outgrow one host.

Named Volumes vs Anonymous Disaster

Named volumes survive docker compose down; anonymous ones don't. Name postgres-db-volume and any DAG volume explicitly so routine restarts keep state. Treat down -v as a data-destruction command that needs a backup first.

Bind mounts connect repo folders into containers: ./dags for code, ./logs for task logs, ./plugins for extensions, plus .env for AIRFLOW_UID. Mounts must appear on scheduler, worker, and webserver identically or each service sees different DAGs.

Back up the DB volume nightly. A pg_dump cron plus a volume snapshot turns a -v disaster from a resume event into a restore drill.

services:
  postgres:
    image: postgres:16
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U airflow"]
      interval: 10s
      retries: 5
    restart: unless-stopped

  airflow-scheduler:
    image: apache/airflow:3.3.1
    command: scheduler
    depends_on:
      postgres:
        condition: service_healthy
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./plugins:/opt/airflow/plugins
    env_file: [.env]
    restart: unless-stopped

volumes:
  postgres-db-volume:  # named: survives `down` without -v
    name: airflow_postgres_db

DAG Sync: Bind Mount vs Git-Sync

Bind mounts win for dev: save a DAG file and the scheduler parses it within 30 seconds. Git-sync sidecars win for staging mirrors: they pull a branch on an interval so the stack tracks version control. Baked images win for prod: DAGs ship inside the image digest with atomic rollback.

Don't mix models on one stack without documenting it. A teammate editing ./dags while git-sync overwrites the same folder creates phantom diffs that waste afternoons.

Whatever you pick, keep DAG delivery in Git. Container-local edits evaporate on recreate and can't be reviewed or rolled back.

Running Migrations Safely

First boot needs an init pass on every OS: docker compose up airflow-init creates the metadata tables and the airflow/airflow login, exiting 0 when done. Seed a reviewable airflow.cfg first with docker compose run airflow-cli airflow config list. Then start the fleet with docker compose up and confirm healthy containers via docker ps.

Migrations must run after every image change and before the scheduler starts. The scheduler crash-looping on a new schema is the classic skipped-migration symptom. Verify with airflow db check, not by watching the UI hopefully. CLI access goes through a service container (docker compose run airflow-worker airflow info) or the airflow.sh wrapper for bash and python shells.

Order startups after reboots: postgres healthy first, then redis, then scheduler and workers. depends_on with service_healthy enforces it; plain depends_on only orders container start, not readiness. Keep AIRFLOW_UID in .env set to your host UID. Without it, container-written logs and DAG files come back root-owned and block the next edit. On SELinux hosts add :z to volume mounts.

# Safe upgrade path: migrate before serving traffic
docker compose pull
# run migrations as a one-shot (matches image version)
docker compose run --rm airflow-cli airflow db migrate
docker compose run --rm airflow-cli airflow db check
# only then restart the long-lived services
docker compose up -d postgres redis
sleep 10
docker compose up -d airflow-scheduler airflow-worker airflow-webserver

docker compose ps --format "table {{.Name}}\t{{.Status}}"

Restart Policies That Survive Reboots

Restart policies decide what survives a host reboot. unless-stopped on scheduler, workers, broker, and postgres brings the fleet back without human clicks. Pair them with healthchecks so Docker restarts genuinely wedged services, not just exited ones.

Startup ordering uses health, not hope. Postgres gets a pg_isready healthcheck; scheduler waits on service_healthy. Redis gets a similar gate before Celery workers subscribe.

Test reboots in staging. Rebooting prod to discover ordering bugs is a career-limiting experiment.

Scaling Services With Compose

Compose scales workers horizontally on one host with docker compose up -d --scale airflow-worker=3. That triples Celery throughput in seconds for morning peaks. It doesn't add hosts, so watch CPU, memory, and DB connections as you scale. Custom needs ride custom images: swap image: for build: ., add a Dockerfile FROM apache/airflow:3.3.1 that pins apache-airflow==${AIRFLOW_VERSION} while pip-installing requirements.txt, then docker compose build or --build on up.

Two shortcuts save afternoons. _PIP_ADDITIONAL_REQUIREMENTS installs small extras at container start without a rebuild, handy for lxml-class additions during iteration. Host-reaching DAGs need extra_hosts: host.docker.internal:host-gateway on the worker plus host.docker.internal URLs, since localhost inside a container isn't your laptop.

Beyond one host's capacity, move to multi-host Celery or Kubernetes. Compose got you to product-market fit; orchestrators carry you past it. Monitor per-worker concurrency as you scale. Three workers at concurrency 8 need 24 DB connections plus headroom, and postgres defaults punish optimists.