Airflow Scheduling: The Catchup That Ran a Month Late

A team enabled a DAG with last month's start_date and went to lunch. They returned to 30 runs hammering the warehouse at once. Nothing was broken. Everything was working as configured.

catchup=True with a past start_date means every missed interval fires. You'll learn why the first run always waits one full interval.

We'll decode schedule values, start_date truth, and data intervals. Timing stops being magic.

What Schedule Actually Means

schedule answers how often, using presets like @daily, cron strings, timedeltas, timetables, or asset lists. It replaced schedule_interval in 3.x and reads far more plainly. One argument states the rhythm.

It does not answer when the first run fires. That comes from start_date plus one interval, a combination that surprises everyone once. Read them as a pair, never alone.

Keep the value boring. Presets for standard cadences, cron for clock times, timetables only for calendars presets cannot express.

Manual runs don't inherit intervals: triggering from UI/CLI/API sets logical_date to now, and data_interval derives from the timetable — don't assume they match. Read logical_date explicitly when a human picks the date. New 3.x CLI is airflow backfill create --dag-id X --from-date ... --to-date ... --max-active-runs 3 --reprocess-behavior failed.

Cron Syntax and Presets

Cron fields read minute, hour, day-of-month, month, day-of-week. 0 6 * fires at 06:00 UTC daily. Presets like @hourly and @daily are aliases for the cron you would otherwise hand-write.

Prefer presets when they match. @daily states intent where 0 0 * states mechanics. Reviewers grasp intent faster and DST discussions stay grounded.

Comment any raw cron with its plain meaning and timezone. Future readers should not parse asterisks under pressure.

import pendulum
from airflow.sdk import dag, task

@dag(
    schedule="0 6 * * *",  # 06:00 UTC daily
    start_date=pendulum.datetime(2026, 9, 1, tz="UTC"),
    catchup=False,
    tags=["reporting"],
)
def morning_report():
    @task
    def build() -> None:
        print("building 06:00 UTC report")

    build()

morning_report()

start_date Is NOT When It First Runs

start_date begins the interval count, not the firing. A daily DAG with a Sept 1 start fires its first run on Sept 2, covering Sept 1 data. The run waits for its interval to complete.

This feels backwards once and obvious forever. Airflow processes complete windows, so the run for a day can only fire after that day ends. Today's partial data is never a complete interval.

Set start_date to a fixed past date with catchup=False for new DAGs. Dynamic start dates like datetime.now() shift the anchor daily and produce ever-growing catchup queues.

Catchup Semantics

catchup=True creates runs for every missed interval since start_date. It is a backfill machine wearing a boolean costume. With a month-old start_date, unpausing fires thirty runs before lunch.

Default new DAGs to catchup=False. The DAG starts from now, history stays unbuilt, and today's data flows first. Rebuild history separately and deliberately.

Use airflow dags backfill with date bounds and run-level limits for history. It paces the rebuild instead of stampeding the warehouse.

Cap the blast with max_active_runs (2-3 on warehouse DAGs) so even intentional catchups queue instead of stampeding. Know the config scheduler.catchup_by_default=False in 3.x — unset catchup means latest-interval-only. Re-enable pauses carefully: turning a DAG off and on re-triggers catchup for the gap.

# preview what catchup will do BEFORE unpausing
AIRFLOW_HOME=~/airflow airflow dags show sales_daily
AIRFLOW_HOME=~/airflow airflow dags list-runs -d sales_daily --limit 5
# deliberate history rebuild under limits (3.x syntax)
AIRFLOW_HOME=~/airflow airflow backfill create --dag-id sales_daily \
  --from-date 2026-08-01 --to-date 2026-08-07 \
  --max-active-runs 3 --reprocess-behavior failed
# cap concurrency in the DAG: max_active_runs=2

Data Intervals: The Mental Model That Fixes Scheduling Confusion

Every run covers a data interval: the logical window its tasks should process. The interval ends at data_interval_end, which is the scheduling anchor tasks must filter by. A Sept 4 run covers Sept 3.

This mental model fixes the classic confusion. The run date is when processing happens; the interval is what data it covers. Late runs still process their own window, not today's.

Write tasks accordingly. Filter sources by interval bounds passed as parameters, never by now(). Reruns then reproduce identical windows.

Schedule in UTC: DST and Timezone Traps

Airflow schedules in UTC, always. A 09:00 local expectation in Berlin fires at 07:00 or 08:00 UTC depending on DST. Twice a year, local-time assumptions breach SLAs by exactly one hour.

Declare start_date with tz=UTC and write schedules as UTC cron. Document the UTC equivalent beside every business-hours promise so reviewers see the mapping.

Test the boundaries. Trigger manual runs across a DST weekend and confirm fire times and interval bounds before the clock does it for you.

timetable for Custom Schedules

Timetables encode calendars presets cannot: business days, holiday skips, blackout windows, multi-asset triggers. They are code, so they test like code.

Most teams never need one. Presets plus cron cover standard cadences honestly, and simpler schedules review faster. Reach for timetables when the calendar has real exceptions.

When you do, enumerate expected run dates in tests across DST and holidays. A timetable without date-list tests is a rumor.

Choose the right timetable class: CronDataIntervalTimetable snaps to the clock (first 22 minutes can vanish if you start at 12:38), DeltaDataIntervalTimetable counts from activation, and CronTriggerTimetable decouples run time from data window (every minute, previous hour). You'll combine Assets with time via AssetOrTimeSchedule when freshness plus a nightly check both matter.

import pendulum
from airflow.sdk import dag, task
from airflow.timetables.calendar import CalendarTimeTable

UTC = pendulum.timezone("UTC")

@dag(
    schedule=CalendarTimeTable(
        "0 6 * * *",  # 06:00 UTC weekdays handled below
        timezone=UTC,
    ),
    start_date=pendulum.datetime(2026, 9, 1, tz="UTC"),
    catchup=False,
    tags=["reporting"],
)
def business_daily():
    @task
    def build() -> None:
        print("weekday build")

    build()

business_daily()