Airflow in Production: Full Pipeline Case Study That Ships

Toy DAGs teach syntax. Production teaches everything else: design docs, idempotent loads, gates that fail loudly, alerts that page, deploys that don't scare you. This guide assembles the whole platform.

You'll follow one company from fragmented cron jobs to a trusted data platform, phase by phase. Each step has exit criteria, owners, and the incident it prevents. Nothing here is theoretical.

By the end you'll hold the senior roadmap: what to build, in what order, and why. Platforms ship. Let's build.

The Company, the Problem, the Constraints

The company ran revenue pipelines on a patchwork: cron jobs calling scripts, warehouse loads owned by whoever wrote them, alerts nobody configured. Month-end turned engineers into firefighters and finance into detectives.

Constraints shaped every choice. A platform team of three, a Snowflake warehouse with real bills, Kubernetes infra, and traffic spiking tenfold at quarter close. No greenfield luxury, no hiring spree coming.

The mandate was trust, not tooling. Business users needed numbers that arrived on time and matched reality. Everything in this guide serves that mandate; technology choices are just implementation details.

Design Doc: DAGs, Schedules, Contracts

The design doc names DAGs, schedules, contracts, and owners on one page. Extract runs hourly into staging, gates validate, the serving DAG promotes daily at 06:00. Row-count bands and uniqueness contracts bind producer to consumer.

Schedules follow business rhythm, not engineering convenience. Hourly extracts absorb source delays, the daily serving run lands before analysts arrive, and backfill windows reserve capacity for month-end replays. Timezones stay UTC everywhere.

Owners sign before building. Data engineering owns extract, analytics owns gate thresholds, platform owns monitoring and deploys. Signatures turn shared concerns into named responsibilities.

Three platform calls were decided up front. Timeouts on every task at ~1.5-2x normal runtime (a stuck task holding a slot forever is a choice, not bad luck). Concurrency in three layers — parallelism for the cluster, max_active_tasks and max_active_runs per DAG — sized together so workers and scheduler agree. And lateness via Deadline, not legacy SLA: Airflow 3 removed classic SLAs, so each family declares reference time plus grace period and the callback pages the owning team.

# design doc: revenue platform v1 (one page, reviewed, signed)
# DAGs:      revenue_extract (hourly) -> revenue_gates -> revenue_serve (daily 06:00)
# Contracts: staging row-count band +-40% of 30d median; zero duplicate (ds, source_id)
# Owners:    extract: data-eng; gates: analytics; alerts: #data-incidents
# Rollback:  redeploy previous image tag; re-promote last good partition
# Exit:      30 green days + one handled incident with runbook update

AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES=7
AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL=90
airflow config get-value dag_processor parsing_processes

Build: Staged by Phase With Code

The build follows the series arc in order. Idempotent extracts land in staging keyed by business date. Gate tasks validate counts and uniqueness. The serving DAG upserts into production tables with SLA callbacks attached.

Each phase exits on proof. Extracts prove rerun convergence with double-run diffs. Gates prove failure on injected bad batches. Monitoring proves paging with a staged scheduler stall. Deploys prove rollback with a rehearsed revert.

Code stays boring on purpose. TaskFlow tasks, Postgres hooks, Slack notifiers: no exotic operators, no clever metaprogramming. Boring code survives on-call rotations; clever code creates them.

Execution rode the Airflow 3 shape: workers never touch the metadata DB directly — they talk through the Task Execution API while scheduler, DAG processor, and API server carry the DB load. DAGs arrived as versioned bundles (GitDagBundle syncing every node, so 'run task X of DAG Y' resolves identically everywhere), which is also what makes DAG versioning traceable per run. Backfills ran scheduler-managed from the UI/API inside concurrency limits, and catchup stayed explicitly False — history only through the targeted command in the runbook.

import datetime
import pendulum
from airflow.sdk import dag, task
from airflow.providers.postgres.hooks.postgres import PostgresHook
from airflow.providers.slack.notifications.slack import send_slack_notification

sla_alert = send_slack_notification(
    slack_conn_id="slack_data_alerts",
    text="SLA miss on {{ dag.dag_id }}: run {{ run_id }} late.",
    channel="#data-incidents",
)

@dag(
    dag_id="revenue_serve",
    schedule="0 6 * * *",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    sla=datetime.timedelta(hours=2),
    sla_miss_callback=sla_alert,
    max_active_tasks=4,
    tags=["tier-1", "revenue"],
)
def revenue_serve():
    @task
    def upsert_partition(ds: str | None = None) -> str:
        hook = PostgresHook(postgres_conn_id="analytics_warehouse")
        hook.run(
            """
            BEGIN;
            DELETE FROM serving.revenue WHERE business_date = %(ds)s;
            INSERT INTO serving.revenue
            SELECT * FROM staging.revenue WHERE business_date = %(ds)s
            ON CONFLICT (business_date, source_id) DO UPDATE
            SET amount = EXCLUDED.amount, status = EXCLUDED.status;
            COMMIT;
            """,
            parameters={"ds": ds},
        )
        return ds or ""

    @task
    def gate_counts(ds: str | None = None) -> str:
        hook = PostgresHook(postgres_conn_id="analytics_warehouse")
        dupes = hook.get_first(
            "SELECT count(*) FROM (SELECT 1 FROM staging.revenue "
            "WHERE business_date = %(ds)s GROUP BY business_date, source_id "
            "HAVING count(*) > 1) d",
            parameters={"ds": ds},
        )[0]
        if dupes:
            raise ValueError(f"dedupe breach: {dupes} duplicated keys")
        return "gates_ok"

    upsert_partition() >> gate_counts()

revenue_serve()

Quality Gates and Monitoring Wiring

Quality gates stand between staging and serving, failing the DAG on breach. Row-count bands catch partial extracts, dedupe queries catch overlapping reruns, null-rate checks catch schema drift. Bad batches quarantine with lineage intact.

Monitoring wires alongside, not after. Heartbeat age pages platform, queue age pages on starvation, SLA misses page the owning team in #data-incidents. Remote logs ship to the bucket from the first run.

Launch criteria are binary. Thirty green days plus one handled incident with a runbook update. The handled incident matters more than the green days: it proves the system pages, humans respond, and learning sticks.

Retention was a decision, not an accident: airflow db clean on a schedule keeping ~30 days of history, so the metadata DB never becomes the bottleneck nobody graphed. Remote logging shipped the same sprint as alerts — on disposable nodes, logs that aren't on S3/GCS (or Elasticsearch/CloudWatch) die with the node, and a post-mortem without logs is a shrug.

CI/CD and Rollout Without Fear

Deploys bake DAGs into versioned images, never sync files mid-write. CI gates prove parse safety, import budgets, and single-run success before any tag ships. Rollback means redeploying the last green tag, a one-command operation.

Promotion moves by environment. Staging runs the candidate image for 48 hours against mirrored schedules; prod takes it only after gates pass and owners sign. Feature DAGs ride separate tags, never the prod release train.

The runbook ships with the deploy. One page per alert with symptom, three commands, escalation, and rollback. On-call reviews it before cutover, so the first page feels like the tenth drill.

CI gated what the scheduler ever saw: lint, DAG parse via DagBag plus airflow dags list-import-errors against the built image (any import error fails the merge, never a 2 AM page), unit tests on callables, and a staging canary before prod promotes. Rollback stayed a redeploy of the last green image — registry keeps 20, so 'what ran when this data went bad' is always answerable. Patch-level Airflow upgrades go zero-downtime when the metadata schema is unchanged; anything touching the schema gets a staged upgrade first.

# image-based deploy: DAGs baked in, never synced mid-write
# docker build -t registry.acme.io/airflow:2026.09.04 .
# docker push registry.acme.io/airflow:2026.09.04
# values: image.tag=2026.09.04, gitSync.enabled=false

# CI gates before any deploy (fail fast, fail loud)
python3 -m pytest tests/test_dagbag.py -q          # DAGs parse, imports under budget
TEST_DAG=revenue_serve airflow dags test revenue_serve 2026-09-01  # single-run proof
# rollback: redeploy previous tag, re-promote last good partition

From Mid-Level to Senior: The Roadmap

Mid-level engineers ship DAGs that work. Seniors ship systems that keep working: design docs that prevent rewrites, idempotent loads that survive reruns, gates that fail loudly, alerts that page, deploys that roll back, runbooks that teach.

Each transition has a teacher. The silent scheduler teaches monitoring, the double-load teaches idempotency, the leaked credential teaches secrets discipline, the broken deploy teaches CI gates. Collect incidents like coursework.

The roadmap never ends. After this platform hums, the next frontier waits: asset-aware scheduling, deferrable everything, tighter contracts. Senior is a direction, not a destination.