Apache Airflow Basics: What Breaks When Cron Runs It

A billing cron ran twice in one night and nobody noticed until customers complained. There was no run history, no retry record, no dependency map. Just a syslog line and an invoice table with twice the rows.

That is the cron trap. It fires commands at times but remembers nothing. When a window is missed or a box reboots mid-job, cron cannot tell you what already ran. You are left reconstructing truth from scattered logs.

Airflow exists to end that guessing. It stores every run, retries failures, and won't run downstream work until upstream work succeeds. You'll see how four components deliver that guarantee.

Four pieces. One source of truth. Zero double-bills.

What a Workflow Engine Does That Cron Can't

Cron fires a command at a wall-clock time and forgets it instantly. There is no record of the run, no retry with backoff, and no notion that step B must wait for step A. If the box reboots mid-job, the next fire has no idea the last one half-finished.

A workflow engine adds memory. Every scheduled run becomes a row in a database with a state, a timestamp, and per-task logs. Failed tasks retry on a policy you define, and downstream tasks wait until their upstreams genuinely succeed.

That memory is what turns a script into a pipeline you can trust at 2 AM. You stop asking did it run and start asking which run, which task, and what did the log say.

Top tutorials also frame the career angle: Airflow shows up in 60%+ of data-engineering listings alongside Spark and dbt. You'll run ETL, ML training, and infra automation on the same engine. It's batch-first though — for sub-second streams you'd pick Kafka or Flink, and you can poll every 30 seconds but that's not real eventing.

The DAG: Tasks, Edges, Acyclicity

A DAG is a Python file that declares tasks and the edges between them. Tasks do the work: run SQL, call APIs, transform files. Edges declare ordering: this task runs after that one, these two run in parallel.

Acyclic is the load-bearing word. The graph cannot loop back on itself, so the scheduler can always sort tasks into a runnable order. If task A waits on B and B waits on A, Airflow refuses the DAG instead of deadlocking at midnight.

You write the what, Airflow handles the when and the what-if. Dependencies, retries, and parallel fan-out come from the graph structure, not from sleep timers in bash.

import pendulum
from airflow.sdk import dag, task

@dag(
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    tags=["billing", "foundations"],
)
def billing_daily():
    @task(retries=3)
    def charge_customers() -> str:
        # idempotent upsert keyed on invoice_id
        return "2026-09-03"

    @task
    def send_receipts(batch_id: str) -> None:
        print(f"receipts for batch {batch_id}")

    send_receipts(charge_customers())

billing_daily()

The 4 Components and How They Talk Through the Metadata DB

Four pieces do the work. The scheduler reads DAG files, creates runs, and queues tasks whose dependencies are met. The executor and its workers run the task instances. The API server and webserver render runs, logs, and controls. The metadata DB sits in the middle as shared truth.

Nothing calls anything directly. The scheduler writes task states to the database, workers claim queued tasks and write results back, and the UI reads the same rows. That indirection is why you can restart any component without losing history.

The triggerer deserves a mention too. Deferred tasks like long-polling sensors park there instead of burning worker slots. It is the reason a 40-minute wait costs nearly nothing in Airflow 3.x.

Pick the executor early. SequentialExecutor is dev-only, LocalExecutor gives single-box parallelism, CeleryExecutor and KubernetesExecutor scale to fleets. The DAG processor parses files and the scheduler creates runs every minute — that's why airflow db migrate (3.x replacement for db init) and one shared Postgres matter before any DAG runs.

# every component reads and writes the same metadata DB
AIRFLOW_HOME=~/airflow airflow db check
AIRFLOW_HOME=~/airflow airflow dags list
AIRFLOW_HOME=~/airflow airflow dags show billing_daily
AIRFLOW_HOME=~/airflow airflow dags list-runs -d billing_daily --limit 10

Idempotency as the First Law

Idempotency means running twice produces the same result as running once. An upsert keyed on invoice_id is idempotent. A blind INSERT of the same rows is not. Every retry, backfill, and manual rerun in Airflow assumes your tasks honor this contract.

The double-billing incident was an idempotency failure wearing a cron costume. The second execution was survivable; the non-idempotent INSERT was not. Had the load been an upsert, the duplicate run would have been a shrug instead of a refund campaign.

Make it a habit now. Key every write on a stable id, partition overwrites by date, and test by clearing a task and re-running it. Row counts must not move.

Airflow's History: Airbnb 2014 to Apache to 3.x

Airflow began inside Airbnb in 2014 as a fix for cron sprawl across data pipelines. It joined the Apache Incubator in 2016 and grew the provider ecosystem that now covers Snowflake, dbt, Kubernetes, and cloud storage.

Airflow 3.x is the current major line. TaskFlow decorators are the default authoring style, the schedule argument replaced schedule_interval, and assets renamed the old dataset triggers. The core promise never changed: tracked runs with atomic tasks.

Knowing the history helps you read old answers. Any tutorial using schedule_interval or execution_date predates 3.x semantics. Translate to schedule and data intervals before copying code.

Airflow 3.x (3.0 April 2025, 3.2.2 May 2026, 2.x EOL April 2026) rebuilt the UI in React, added built-in DAG versioning so code edits don't disturb in-flight runs, renamed Datasets to Assets for event-driven triggers, and locked tasks behind a Task Execution API instead of direct DB access. Imports consolidate to airflow.sdk and schedule replaces schedule_interval.

import pendulum
from airflow.sdk import dag, task
from airflow.providers.postgres.hooks.postgres import PostgresHook

@dag(
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    tags=["billing"],
)
def invoice_upsert():
    @task(retries=3)
    def upsert_invoices(batch_id: str) -> None:
        hook = PostgresHook(postgres_conn_id="billing_db")
        hook.run(
            """
            INSERT INTO invoices (invoice_id, customer_id, amount, batch_id)
            VALUES (%(invoice_id)s, %(customer_id)s, %(amount)s, %(batch_id)s)
            ON CONFLICT (invoice_id) DO UPDATE
            SET amount = EXCLUDED.amount, batch_id = EXCLUDED.batch_id
            """,
            parameters={
                "invoice_id": "inv-88412",
                "customer_id": "cust-2091",
                "amount": 4900,
                "batch_id": batch_id,
            },
        )

    upsert_invoices("2026-09-03")

invoice_upsert()

When Airflow Is (and Isn't) the Right Tool

Airflow shines for scheduled batch work with dependencies: nightly ETL, hourly aggregates, weekly reports. If runs need history, retries, and a UI, it earns its operational cost quickly.

It is the wrong tool for event streaming, sub-second latency, or a single cron line that never fails. Kafka and Flink own streams; cron owns trivial timers. Running a five-second health check through a full Airflow stack is theater.

The honest test is state. If you need to answer what ran, what failed, and what happens on rerun, choose Airflow. If those questions never come up, keep the simpler tool.

Versus Prefect and Dagster: Airflow wins on 80k+ orgs, 30M monthly downloads, and hundreds of providers for Snowflake, dbt, and clouds. Prefect feels lighter for Python-first teams, Dagster models assets natively. Choose Airflow when you need proven scheduling, a deep provider bench, and Grid/Graph observability at 2 AM.