Airflow Backfill: Taming the 90-Day Catchup Storm Safely

Every Airflow engineer learns the catchup lesson the hard way: a DAG deployed with a start_date six months in the past and catchup=True, and the scheduler suddenly queues 180 runs that hammer the database, the API, and the warehouse simultaneously.

Catchup isn't evil — it's a mechanism. The problem is deploying it by accident and running it without limits.

Here's what catchup actually does, the backfill CLI that gives you control, safe rerun patterns, and the production checklist that keeps a backfill from becoming an incident.

1. What Catchup Actually Does

Airflow schedules one DAG run per data interval between start_date and now. Catchup is the setting that decides what happens to intervals you missed while the DAG was paused, undeployed, or failing.

With catchup=True, the scheduler queues runs for every missed interval as fast as it can. With catchup=False, it skips the missed intervals entirely and runs only the most recent one.

Modern Airflow defaults to catchup=False, but the setting can be flipped per DAG — and that's where incidents come from.

2. The Backfill CLI

The backfill CLI gives you manual control over missed intervals: an explicit date range, a concurrency limit, and a reset flag for reruns.

The core command targets a range and caps concurrent runs. Add --reset-db-runs to re-run intervals that already have run records — the command that turns "redo the failed ones" into one line.

The range semantics matter: --start-date is inclusive, --end-date is exclusive. Mistake the boundary and you either re-run a day you didn't want or skip the day you needed.

Airflow 3 renamed the command and added flags: airflow backfill create --dag-id market_etl --from-date 2026-03-01 --to-date 2026-03-31 --reprocess-behavior failed --max-active-runs 4 --run-backwards — --reprocess-behavior picks which existing runs to redo, and --run-backwards processes the range in reverse order, most recent interval first. The same form exists in the UI: DAG Details → Trigger → Backfill.

# Backfill March only, max 4 concurrent runs
airflow dags backfill market_etl \
  --start-date 2026-03-01 \
  --end-date 2026-03-31 \
  --max-active-runs 4

# Re-run intervals that already ran (after a bug fix)
airflow dags backfill market_etl \
  --start-date 2026-03-01 \
  --end-date 2026-03-31 \
  --reset-db-runs \
  --max-active-runs 2

# See which runs a backfill created
airflow dags list-runs -d market_etl

3. Clearing Task Instances Safely

Clearing is the surgical rerun tool: airflow tasks clear resets task instances to their initial state so the scheduler re-queues them.

Scope it narrowly. Clear one task across a date range, or one interval, or one task instance in the grid view. The common production mistake is clearing with a wide range and letting everything re-run — including tasks that never failed.

Clearing does not delete data. It resets Airflow's bookkeeping. Whether the re-run double-loads the warehouse depends entirely on whether your tasks are idempotent.

The mechanics are precise: clearing sets the task instance's state back to None and resets max_tries to 0, which forces the scheduler to re-queue it — and the Grid view preserves the previous attempts as task instance history. Scope flags exist for every rerun shape: --past, --future, --upstream, --downstream, --recursive, and --failed, which re-runs only the instances that actually failed.

# Clear one task across a date range (re-run just that task)
airflow tasks clear market_etl --task-id load_market_data \
  --start-date 2026-03-01 --end-date 2026-03-15

# Clear all tasks for one interval
airflow tasks clear market_etl \
  --start-date 2026-03-08 --end-date 2026-03-08

# Clear a failed task instance and nothing else (UI equivalent)
airflow tasks clear market_etl --task-id hit_polygon_api --downstream

4. Reruns Without Double Loads

Every rerun story ends at the same question: will running this task again produce the same data, or duplicate it?

The answer is idempotency. A load task that inserts with no key check duplicates on rerun. A load task that upserts by a natural key — or deletes the interval's partition first — reproduces the same rows.

The production pattern for date-partitioned loads: delete the partition for the data interval, then insert. Rerun the backfill as many times as you like; the data is identical.

from airflow.decorators import task
from airflow.providers.postgres.hooks.postgres import PostgresHook


@task
# Idempotent: delete the interval's partition, then insert
def load_market_data(rows: list[dict], interval_end: str):
    hook = PostgresHook(postgres_conn_id="warehouse")
    with hook.get_conn() as conn:
        with conn.cursor() as cur:
            cur.execute(
                "DELETE FROM market_data WHERE interval_date = %s", (interval_end,)
            )
            for row in rows:
                cur.execute(
                    "INSERT INTO market_data (symbol, interval_date, close) VALUES (%s, %s, %s)",
                    (row["symbol"], interval_end, row["close"]),
                )

5. Scheduled vs Manually Triggered Runs

The grid view distinguishes scheduled runs from manual runs, and the distinction matters for backfills.

Scheduled runs carry the data interval in their run metadata. Manual triggers (the Trigger DAG button, the API, the CLI) also get a data interval — the next one — which is why a manual run during an active interval can confuse people: it runs the same interval as the scheduler is about to run.

The backfill CLI creates runs with their own intervals, marked as backfill runs. When you look at the grid, backfill runs and scheduled runs are different entries with the same interval — which is exactly why duplicate data protection (idempotent loads) matters again.

6. The Production Backfill Checklist

A backfill is a controlled operation. Before running one on production, walk this checklist.

1. Confirm the loads are idempotent — test the rerun on a staging interval.
2. Set catchup=False on the DAG so the scheduler doesn't race you.
3. Pick an explicit range, inclusive start, exclusive end.
4. Cap concurrency: --max-active-runs 2-4 for warehouse loads.
5. Monitor the grid view and the warehouse's load; throttle if queries queue.
6. After completion, compare row counts against the source per interval.
7. Document the backfill in the runbook: range, duration, row counts.

# Step 3-4: the disciplined backfill
airflow dags backfill market_etl \
  --start-date 2026-01-01 \
  --end-date 2026-03-31 \
  --max-active-runs 4

# Step 6: per-interval verification
SELECT interval_date, count(*) AS rows
FROM market_data
WHERE interval_date BETWEEN '2026-01-01' AND '2026-03-30'
GROUP BY interval_date
ORDER BY interval_date;