Apache Airflow Basics: What Breaks When Cron Runs It

Cron is the most dangerous scheduler in your stack, and it's not close. It fires commands on schedule and forgets them instantly — no state, no retries, no dependencies, no idea what already ran. That amnesia cost a billing team their customers' trust when a missed window silently re-ran the monthly invoicing job and double-billed everyone.

A workflow engine exists to remember. Apache Airflow schedules directed acyclic graphs (DAGs) of tasks, tracks every run in a metadata database, retries failures, and makes each execution auditable. Four components — webserver, scheduler, executor, metadata DB — cooperate through that one database, which is why losing it feels like losing the whole system.

This article builds the mental model from the failure up: what cron can't do, what a DAG is, how the components talk, and why idempotency is the first law of every pipeline you'll write. Then it gets honest about when Airflow is the answer, and when it's a sledgehammer for a nail.

Cron never learns from its mistakes. You're about to use a tool that does.

1. What a Workflow Engine Does That Cron Can't

Cron fires commands on a schedule. That's the whole contract: minute, hour, day, command. It has no memory of what ran, no notion of success beyond the exit code, and no way to express "run this only after that finished."

A workflow engine adds the three things cron never had: state, dependencies, and retries. Every execution becomes a tracked run with an ID. Every task records a transition from queued to running to success or failure. When a task fails, the engine retries it — with backoff, without a human paging at 3 AM.

The billing incident was a state problem. Cron re-ran a job over data it had already processed because nothing recorded that it had. Airflow's metadata DB is that memory.

# /etc/crontab — the billing job that double-billed
0 2 1 * * root /opt/billing/run_monthly_billing.sh

# After cron fires it, the system remembers one thing: an exit code.
# No run ID. No state. No retry. No dependency. No audit trail.

2. The DAG: Tasks, Edges, and Acyclicity

A DAG is a directed acyclic graph: nodes are tasks, edges are dependencies. "Acyclic" is the non-negotiable part — no task may depend on itself through a cycle, because the scheduler couldn't decide where to start. Airflow parses your file and refuses cycles at import time.

Tasks are units of work: pull an API, write a file, run SQL, send email. Edges express order: extract >> transform >> load reads as "extract, then transform, then load." The graph is the pipeline's documentation — the Grid view shows every run's graph with states colored per task.

A cron job is one point in time. A DAG is a structure: branching, joining, retrying, and failing in ways that stay visible.

Every task needs a task_id and an owner — a fresh install defaults the owner to airflow, and Airflow raises an error if either is missing. Every DAG run also carries a data interval: the slice of data (and time) the run operates on, which is what makes reruns and backfills addressable. And use timezone-aware datetimes (pendulum) for start_date — the Python stdlib timezone has known limitations in Airflow.

from datetime import datetime

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator


def generate_invoice(customer_id: str) -> dict:
    return {"customer_id": customer_id, "amount_usd": 49.99}


with DAG(
    dag_id="monthly_billing",
    schedule="@monthly",
    start_date=datetime(2026, 1, 1),
    catchup=False,
) as dag:
    extract_customers = BashOperator(
        task_id="extract_customers",
        bash_command="python /opt/billing/extract_customers.py",
    )
    generate_invoice_task = PythonOperator(
        task_id="generate_invoice",
        python_callable=generate_invoice,
        op_kwargs={"customer_id": "cust_1042"},
    )
    extract_customers >> generate_invoice_task

3. The Four Components and the Metadata DB

Airflow ships four processes. The webserver renders the UI and the REST API. The scheduler decides what should run and when. The executor — a class configured per deployment — claims ready tasks and runs them. The metadata database persists everything: DAG definitions, DAG runs, task states, XComs, connections.

The trick is that no component talks to another directly. The scheduler writes "this task is queued" into the DB; the executor reads it, runs the work, and writes back the outcome; the webserver reads the same rows for the UI. The DB is the single source of truth — which is why losing it looks like losing the whole system.

SQLite ships by default and is fine for learning. Production runs Postgres; every real deployment I've seen graduates to it within weeks.

4. Idempotency as the First Law

Run the same DAG twice and the world should look identical. That's idempotency, and it's the property that makes retries, backfills, and catchup safe. A retry is a re-execution of a task — if that task appends to a table, you now have double rows.

Design for it: upsert by natural key, delete-then-insert per partition, or skip rows already processed. The billing DAG's fix was a dedupe key — an invoice row only existed once per customer-month.

Airflow will re-run your tasks. Assume it. Build every task as if it will execute twice, because it will.

5. Airflow's History: Airbnb 2014 to Apache 3.x

Airflow started at Airbnb in 2014 as an internal tool for the company's growing data pipelines. It hit the Apache Incubator in 2016 and became a top-level Apache project in 2019. The 1.x era made the scheduler, the DAG authoring model, and the UI.

2.0 (2020) rewrote the scheduler, added the TaskFlow API and the Grid view. 3.x is the current major — new operator module paths like airflow.operators.bash, assets replacing datasets, the triggerer, and object storage support.

Why does history matter? You'll meet 1.x-era DAGs in production codebases: default_args dicts, schedule_interval, imports from airflow.operators.bash_operator. The core model hasn't changed since 2014 — DAGs, tasks, states, a scheduler, a DB. That stability is why the platform still runs a huge share of the world's batch pipelines.

6. When Airflow Is — and Isn't — the Right Tool

Airflow wins when work has dependencies, needs retries, and must be visible: batch ETL, data platform orchestration, pipeline fan-in and fan-out. It loses when you need millisecond real-time triggers, long-running services, or a single nightly script that rarely fails.

If your problem is "one script, runs nightly, fails rarely" — cron is fine and honest. If it's "20 jobs, 5 depend on each other, failures must page someone" — that's Airflow's floor, not its ceiling.

The honest test: can you draw it as a DAG with tasks? If yes, Airflow is probably the right answer. If the graph would be one node, you're paying for a platform to run a shell script.