Airflow Backfill: 90-Day Catchup Storm Tamed in Prod

Nobody fears the scheduler until it does exactly what they asked. You deploy a daily DAG on a Monday, glance at the Grid view, and find 90 queued runs staring back at you.

That's catchup doing its job. It saw 90 missed intervals between your start_date and today and queued every single one. Your workers, your warehouse, and your on-call rotation are about to have a very bad morning.

There's a calmer path. Explicit backfills with capped concurrency rebuild exactly the dates you name and nothing else. You'll learn both mechanisms here.

Catchup Versus Backfill: Know the Difference

Catchup is a scheduler behavior, not a command. When a DAG has catchup=True, the scheduler compares start_date to now on every parse and creates a run for each missed interval. A daily DAG with a 90-day-old start_date creates 90 runs on its first heartbeat. That's the trap: the default looks harmless in dev and explodes in prod.

Backfill is the deliberate alternative. You name the DAG, the date range, and the concurrency cap, and Airflow creates exactly those runs. Airflow 3.x uses airflow backfill create --from-date/--to-date with --max-active-runs and --reprocess-behavior. Airflow 2.x uses airflow dags backfill --start-date/--end-date. Both funnel through the scheduler, but only backfill lets you dry-run first.

The rule you'll keep: catchup=False everywhere in production. History becomes a conscious command with a cap, not a deploy side effect that pages you at 2 AM.

Catchup Mechanics and the Default Trap

This DAG shows the safe production shape. catchup=False stops the scheduler from inventing history. max_active_runs=3 bounds how many runs compete for workers at once. The load task deletes its own partition before inserting, so replaying a date changes zero net rows.

Notice the pool assignment. Backfill runs share pools with scheduled runs, so isolating heavy loads in warehouse_pool keeps a rebuild from starving lightweight hourly DAGs. You'll thank yourself the first time a backfill overlaps the morning peak.

Tags look trivial until you run list-runs across 100 DAGs. Tag every prod DAG so your backfill audit commands can filter fast.

from datetime import datetime, timedelta
from airflow.sdk import dag, task

@dag(
    dag_id="sales_daily",
    schedule="@daily",
    start_date=datetime(2026, 5, 1),
    catchup=False,  # prod default: never auto-replay history
    max_active_runs=3,  # caps scheduled + backfill pressure
    default_args={"retries": 2, "retry_delay": timedelta(minutes=5)},
    tags=["sales", "prod"],
)
def sales_daily():
    @task(pool="warehouse_pool")
    def load_day(logical_date=None):
        ds = logical_date.strftime("%Y-%m-%d")
        # idempotent: delete partition, then insert
        return f"DELETE FROM sales WHERE ds='{ds}'; INSERT INTO sales SELECT * FROM staging WHERE ds='{ds}'"
    load_day()

sales_daily()

Backfill CLI: Dates, Reruns, and Limits

Airflow 3.x made backfills first-class: they are scheduled and tracked like regular runs with UI and API support. The create command takes --from-date and --to-date (inclusive), --max-active-runs for this backfill only, --reprocess-behavior (none, failed, completed), --run-backwards for newest-first ordering, and --dag-run-conf '{"my": "param"}' to inject run config. --dry-run prints dates without creating anything. The same rebuild exists in the UI: open the DAG details page, hit Trigger, pick Backfill, and choose Missing Runs, Missing and Errored Runs, or All Runs with an optional JSON conf. Backfills also work through the REST API for scripted rebuilds.

Reprocess behavior decides what happens when a run already exists. none skips existing runs entirely. failed creates new runs only where the prior run failed. completed re-creates runs that finished or failed. If the latest run is still queued or running, Airflow creates nothing for that date no matter which behavior you pick. Pick failed for incident recovery and completed only when you distrust old outputs. Backfill max_active_runs applies independently of the DAG-level max_active_runs, so the two caps stack.

Two edge cases bite teams. Backfill makes no sense for DAGs without a time-based schedule, so don't fire it at @once or asset-only DAGs. Partitioned-timetable DAGs (CronPartitionTimetable) reuse the same --from-date/--to-date flags, with Airflow interpreting the range as partitions and creating one run per partition. Run one date first, diff row counts against the warehouse, then widen the window. Backfills replay code against current tables, so a schema change since June can break a May rerun in ways scheduled runs never show.

Clearing Task Instances Safely

Clearing resets task instances to queued so they run again. That's powerful and dangerous in equal measure. A bare airflow tasks clear with no dates resets the whole DAG history, including months of green runs your finance team already reconciled.

Scope every clear three ways: the DAG id, a date window, and a task regex. The regex is the part people skip. Clearing load_day tasks is safe when extracts are expensive; clearing extracts too means re-pulling APIs you already paid for.

After clearing, watch one date go green before walking away. Grid view shows task instance history per cell, so you can confirm the retry actually replaced the failure instead of stacking beside it.

# Airflow 3.x: targeted, capped, resumable
# 1) preview the window (creates nothing)
airflow backfill create --dag-id sales_daily \
  --from-date 2026-06-01 --to-date 2026-06-10 \
  --dry-run

# 2) rebuild only failed dates, 2 at a time
AIRFLOW_CTX=prod airflow backfill create --dag-id sales_daily \
  --from-date 2026-06-01 --to-date 2026-06-10 \
  --reprocess-behavior failed --max-active-runs 2

# 3) legacy 2.x spelling (same idea, older flags)
# airflow dags backfill --start-date 2026-06-01 --end-date 2026-06-10 sales_daily

# 4) track progress
airflow dags list-runs sales_daily -s 2026-06-01 -e 2026-06-10 -o table

Scheduled vs Manually Triggered Runs

Scheduled runs move forward; manual runs and backfills rewrite the past. The scheduler paces scheduled runs by max_active_runs and pool slots. Backfills add extra runs alongside them, which is why prod slows during big rebuilds even though nothing is broken.

Manually triggered runs use the current code and current data, not the historical context. That's fine for testing but wrong for rebuilding June with June's logic. Where your version supports it, backfill against the DAG version that originally ran so history replays faithfully.

If downstream DAGs consume your outputs, tell their owners before you backfill. Rewritten partitions change row counts downstream dashboards already published, and nobody likes explaining a restated metric on a Friday.

Rerun Without Double-Loads

Idempotency is what makes reruns boring. A task keyed on its logical_date partition can run five times and leave the table identical. An append-only task doubles revenue every time you touch it. Guess which one pages finance.

The pattern is simple: derive ds from logical_date, delete that partition, then insert exactly that partition's rows. Row-count checks after the load catch grain changes before downstream sees them. Dedupe keys catch the rest.

Backfill day is the wrong day to discover your loads append. Test reruns in staging with --reprocess-behavior completed and assert zero net change. If the count moves, don't backfill prod yet.

Backfill on Prod: The Checklist

Big rebuilds are capacity events. Announce the range, the expected duration, and who to ping if SLAs slip. Pause only when catchup is actively flooding; otherwise leave the schedule running so today stays green while history rebuilds beside it.

Stagger ranges longer than two weeks into weekly chunks with verification between them. Each chunk gets a row-count query per ds before the next starts. That cadence caught a grain change on day 9 of a 30-day rebuild in one real incident, saving 21 days of bad rewrites.

Keep the concurrency low until the first chunk drains cleanly. Raising --max-active-runs from 2 to 4 is easy; explaining missed SLAs to three teams is not.

# Prod backfill checklist (run in order)
# 0) announce: who is affected, what range, rollback plan
airflow dags list-runs sales_daily -s 2026-07-20 -e 2026-08-02 -o table

# 1) pause only if catchup is still flooding the queue
# airflow dags pause sales_daily

# 2) dry-run the exact window
airflow backfill create --dag-id sales_daily \
  --from-date 2026-07-20 --to-date 2026-08-02 --dry-run

# 3) capped rebuild of failed dates only
airflow backfill create --dag-id sales_daily \
  --from-date 2026-07-20 --to-date 2026-08-02 \
  --reprocess-behavior failed --max-active-runs 2

# 4) verify row counts per ds, then unpause
# SELECT ds, COUNT(*) FROM sales WHERE ds BETWEEN '2026-07-20' AND '2026-08-02' GROUP BY ds;