Airflow Scheduling: The Catchup That Ran a Month Late

Everyone's first Airflow DAG uses a cron string and a start_date, and everyone gets burned by the exact same gap. The schedule doesn't fire at start_date. It fires after each data interval closes, and catchup decides whether every missed interval from start_date to now runs at once. That gap is where pipelines go wrong.

We deployed a daily revenue DAG with a start_date set a month in the past "to be safe." The scheduler read that as a month of unpaid bills and fired 30 runs in the first few minutes. Nobody had misconfigured a cron. Everybody had misunderstood what a schedule actually means.

This article walks the scheduling stack from cron syntax to custom timetables, and builds the data-interval mental model that fixes the confusion for good. Get the interval right and scheduling stops being a guessing game. Get it wrong and you'll get a month-late DAG of your own.

1. What "Schedule" Actually Means

A schedule is not a timer. It's a generator of data intervals. Airflow asks the schedule "what intervals exist," then creates one run per interval, and each run processes the window that just closed.

That distinction is the whole ballgame. A cron fires at a timestamp; an Airflow schedule fires a run that owns a window of time. The window is called the data interval, and it's why two DAGs with the same cron can still mean completely different things.

Once you accept that runs process intervals, everything else — start_date, catchup, backfills — clicks into place. The clock is only the trigger. The interval is the contract.

2. Cron Syntax and Presets

Airflow accepts standard five-field cron: minute, hour, day of month, month, day of week. The trap is the field order — minute comes first, and the most common mistake is writing "0 6 *" and believing it means 6 minutes past midnight. It means 06:00.

Presets are sugar over the same five fields. @daily is "0 0 ", @hourly is "0 *", and @weekly, @monthly, @yearly follow the same pattern. @once schedules a single run, and @continuous runs back-to-back intervals for streaming-ish workloads.

Use presets until you need minute-level control. A team reading "@daily" instantly knows the cadence; a team reading "17 3 1-5" needs a man page.

Two more options round out the menu: a datetime.timedelta object schedules by elapsed time with no cron at all, and Airflow parses cron through the croniter library, so extended syntax like */15 and day-of-week lists works as expected. One preset carries a caveat: @continuous fires back-to-back intervals, and the docs pair it with max_active_runs=1 so runs never overlap.

from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator

with DAG(
    dag_id="daily_revenue_rollup",
    schedule="0 6 * * *",              # 06:00 UTC, every day
    start_date=datetime(2026, 8, 1),
    catchup=False,
    tags=["billing", "daily"],
) as dag:
    def rollup(**context):
        interval_start = context["data_interval_start"]
        print(f"Rolling up revenue for {interval_start.isoformat()}")

    rollup_task = PythonOperator(
        task_id="rollup_revenue",
        python_callable=rollup,
    )

3. start_date Is NOT When It First Runs

Here's the sentence that fixes most scheduling confusion: the first run fires after start_date plus one full interval, never at start_date.

Set a @daily DAG with start_date of August 1, and the first run fires on August 2 at 00:00, processing August 1's data. The start_date anchors the first data interval, not the first execution. Airflow waits for that interval to end before it has anything to process.

This is also why "start_date yesterday" feels like a safe default and isn't. With catchup off it mostly is harmless, but with catchup on, a start_date in the past means a backlog you never asked for. Pick start_date as exactly one interval before the first data you want processed.

4. Catchup Semantics: The Month-Late Incident, Deep Dive

Catchup answers one question: what do we do with the intervals between start_date and now that never ran? With catchup=False, nothing — the scheduler picks up from the next interval and moves forward. With catchup=True, every missed interval fires, all of them, as fast as the scheduler can queue them.

Our revenue DAG had a start_date one month back and catchup=True. Thirty missed intervals became thirty queued runs in minutes. The scheduler didn't stagger them over thirty days; it reconciled the ledger instantly. Runs weren't late by a month — they all happened at once.

The fix had three parts: catchup=False in the DAG, start_date moved to go-live, and explicit backfills with airflow dags backfill --start-date --end-date whenever history was actually needed. Backfill became an intentional act, not an accident of deployment.

Two semantics make backfill safe to use: the scheduler executes backfilled runs sequentially, one after another, and each run is atomic and idempotent — the properties that make re-running any interval harmless. There's also a fleet-wide default: catchup_by_default in airflow.cfg sets the catchup value for every DAG that doesn't declare one, so the global default can be flipped instead of auditing each file.

from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator

with DAG(
    dag_id="daily_revenue_rollup",
    schedule="@daily",
    start_date=datetime(2026, 8, 1),
    catchup=False,          # never auto-backfill missed intervals
    tags=["billing"],
) as dag:
    def aggregate(**context):
        start = context["data_interval_start"]
        end = context["data_interval_end"]
        print(f"Aggregating revenue for {start} -> {end}")

    aggregate_task = PythonOperator(
        task_id="aggregate_revenue",
        python_callable=aggregate,
    )

5. Data Intervals: The Mental Model That Fixes Scheduling Confusion

Stop thinking in execution dates. Think in data intervals. A run's identity isn't when it ran — it's the window of data it processed: data_interval_start to data_interval_end.

For a @daily DAG, the run that fires at midnight on August 2 processes August 1 00:00 to August 2 00:00. The scheduler fires on the interval end, because that's when the data for the interval is complete. Airflow's legacy term execution_date is really just data_interval_start — same value, worse name.

Every task gets both values through context, and production tasks should use them constantly: log them, filter on them, name output files with them. A task that knows its interval is a task that survives reruns, backfills, and late schedules.

Version note for older code: the execution_date concept was deprecated in Airflow 2.2, which is why every current doc says data_interval_start instead.

from airflow.decorators import dag, task
from airflow.utils.dates import days_ago

@dag(
    schedule="@daily",
    start_date=days_ago(2),
    catchup=False,
    default_args={"retries": 2},
)
def nightly_orders():
    @task
    def count_orders(data_interval_start, data_interval_end):
        print(f"Interval: {data_interval_start} -> {data_interval_end}")
        return f"count_orders_{data_interval_start:%Y%m%d}"

    count_orders()

nightly_orders()

6. Schedule in UTC: DST and Timezone Traps

Airflow defaults to UTC, and that default is a feature. Cron in UTC means the schedule never blinks when daylight saving rolls around. The same expression in a local timezone fires one hour early in spring and one hour late in autumn — and a poorly timed DST boundary can even skip or double-fire a run.

The rule: schedules live in UTC wall-clock time. "0 6 *" means 06:00 UTC, full stop. If the business wants 06:00 Paris time, compute the UTC offset once and put it in the schedule, or do timezone translation inside tasks where it belongs.

Set default_timezone=utc in airflow.cfg and never let a local timezone sneak into a schedule string. Your pipeline doesn't care what the office clock says.

7. Timetables for Custom Schedules

Cron expresses regular calendars well and almost everything else badly. Trading-hours schedules, quarter-end runs, fiscal weeks, skip-the-holiday logic — all of it is a Timetable's job.

A Timetable is a class that answers two questions: given a time, what interval does it belong to (infer_data_interval), and given the last run, what's the next one (next_dagrun_info). Register it as a plugin, reference it from schedule=, and you've replaced cron with real logic.

The beauty is testability. A Timetable is a plain class — you can unit-test its interval math without a running scheduler. If your schedule is worth writing, it's worth testing, and cron strings can't be tested at all.

The interface expects two methods: next_dagrun_info (the next scheduled interval) and infer_manual_data_interval (what a manual trigger's interval should be), both returning a DataInterval with a start and an end.

from airflow.plugins_manager import AirflowPlugin
from airflow.timetables.base import DagRunInfo, DataInterval, Timetable

class MarketHoursTimetable(Timetable):
    def infer_data_interval(self, run_after):
        start = run_after.set(hour=9, minute=30)
        return DataInterval(start=start, end=start.add(days=1))

    def next_dagrun_info(self, last_automated_data_interval, restriction):
        next_start = last_automated_data_interval.end.add(days=1)
        return DagRunInfo.interval(DataInterval(next_start, next_start.add(days=1)))

class MarketHoursPlugin(AirflowPlugin):
    name = "market_hours_timetable"
    timetables = [MarketHoursTimetable]