Airflow Docker Compose: The Setup That Wiped Our DAGs

The official docker-compose.yaml is the fastest way to run Airflow anywhere. One file, eight services, one command, and you're up. It's also the fastest way to lose everything if you don't understand its volumes.

The stack is designed around state: the Postgres metadata DB and the DAGs folder are the two things you can never afford to lose. Both live in volumes, and the difference between named and anonymous volumes decides whether they survive cleanup.

Most teams learn this the hard way. A hung scheduler, a frustrated engineer, and the muscle-memory command docker compose down -v. Minutes later the stack is back up: empty, clean, and newly deployed DAGs.

Treat the compose file as infrastructure you own, not a script you run. Your DAGs depend on it.

1. The Official Compose File, Service by Service

The official docker-compose.yaml runs Airflow as one stack of services. postgres holds the metadata DB, redis is the Celery broker, webserver serves the UI, scheduler parses DAGs, worker executes tasks, and airflow-init runs the database migrations once on startup.

Each service uses the same Airflow image with a different entrypoint command. That's the whole trick: one image, six roles, distinguished by command and dependency order.

airflow-init is the quiet hero. It runs airflow db migrate before anything else, creating the metadata schema. Skip it and webserver greets you with 500s and scheduler dies on missing tables.

Current Airflow 3 compose files split the roles further: airflow-dag-processor parses DAG files, airflow-api-server serves the UI (the old webserver role), and airflow-triggerer runs the event loop for deferrable operators. The images are pinned deliberately — postgres:16 and redis:7.2-bookworm (Redis changed its license after 7.2). Flower, the Celery monitor on port 5555, rides along behind a profile: docker compose --profile flower up.

Two environment notes before first boot. On Linux, export AIRFLOW_UID=$(id -u) — the default 50000 makes the init container chown your dags and plugins folders to a user you don't own. And give Docker at least 4 GB of RAM for the Airflow containers; the init service warns when it detects less.

Understand the roles before you touch a single flag. Every debugging session starts here.

2. Named Volumes vs Anonymous: Where State Lives

Volumes are where container state survives. Anonymous volumes get a random name at creation; named volumes get the name you give them. That one difference decides what survives docker compose down -v.

The metadata DB must survive restarts and cleanups. Declare postgres-db-volume and mount it at the Postgres data directory. Losing it means losing run history, variables, connections, and every audit trail.

The same goes for the DAGs folder when you mount it from a volume: schedule it as scheduler-dags-volume, or you'll wonder where your DAGs went after the first cleanup.

Name your volumes. Anonymous volumes are a delete waiting to happen.

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data

volumes:
  postgres-db-volume:
    name: airflow-postgres-db
  scheduler-dags-volume:
    name: airflow-dags

3. DAGs Sync: Bind Mount vs git-sync

Your DAGs need to reach the scheduler and workers. The bind mount is the dev workhorse: ./dags on your host maps into /opt/airflow/dags in the containers. Edit on the host, see it in the UI within a parse cycle.

Mount it read-only in production. A writable mount lets the container mutate your source, and you'll chase phantom diffs.

git-sync is the production pattern: a sidecar container clones a repo branch into the DAGs folder on an interval. Merge to main, and DAGs land without touching the host.

The rule: bind mount for fast iteration, git-sync for anything that needs a review trail.

services:
  scheduler:
    image: apache/airflow:3.3.0
    volumes:
      - ./dags:/opt/airflow/dags:ro
      - ./logs:/opt/airflow/logs
      - ./config:/opt/airflow/config
    environment:
      AIRFLOW__CORE__DAGS_FOLDER: /opt/airflow/dags

  worker:
    image: apache/airflow:3.3.0
    volumes:
      - ./dags:/opt/airflow/dags:ro

4. Running Migrations Safely

The metadata schema must match the Airflow version. The init service runs airflow db migrate on startup, applying migrations in order. That's the safe path.

The unsafe path is running migrations ad hoc while webserver and scheduler are already serving. Two containers migrating the same schema race, and you get lock errors and half-applied versions.

Before you change anything destructive, take a dump. pg_dump the metadata DB to a file, label it with the date, and store it somewhere outside the compose stack. Restore is one command.

Migrations are boring until they're not. Back up first, migrate once, verify after.

# Back up the metadata DB before touching anything destructive
docker compose exec postgres pg_dump -U airflow -d airflow \
  > backups/airflow_$(date +%F).dump

# Verify the dump is readable
ls -la backups/airflow_$(date +%F).dump

# Restore when the unthinkable happens
cat backups/airflow_2026-08-01.dump | \
  docker compose exec -T postgres psql -U airflow -d airflow

5. Restart Policies and Crash Loops

Compose gives every service a restart policy. unless-stopped is the production default: the container comes back after crashes and host reboots, but respects an explicit stop. Set it as a service key, as in restart: unless-stopped on scheduler, webserver, and worker.

Set it and a transient OOM or a host reboot stops taking your platform down with it. Skip it and every reboot becomes a manual docker compose up.

A restart loop tells you something real: config errors, missing volumes, or a bad image tag. Watch the logs, not the uptime counter. If the scheduler restarts every 5 seconds, it's not the policy's fault.

The policy is a safety net, not a fix. Diagnose the loop, then let the policy keep the healthy stack alive.

6. Scaling Services with --scale

Need more workers? Compose scales a service with one flag: docker compose up -d --scale worker=3. Each replica runs the same image and config, pulling from the same metadata DB and broker.

Celery workers are the safe thing to scale in the compose stack. They're stateless: the queue hands them work, they report results to the DB. More workers, more concurrency, no state to sync.

What you should never scale: postgres, webserver, or scheduler behind a single Compose deployment. Multiple schedulers need HA configuration, and multiple webservers need a shared config story. That's a production upgrade, not a flag.

Scale the stateless workers. Leave the stateful services alone.

# Scale the Celery worker fleet to 3 replicas
docker compose up -d --scale worker=3

# Confirm replicas are running
docker compose ps

# Check worker heartbeats
docker compose logs worker | tail -20