Airflow Performance: Scheduler Stalls at 2,000 DAGs

Schedulers don't complain when they're drowning. They just schedule a little later each loop until your 6 AM DAG starts running at lunch. You'll blame the workers first. Don't.

One platform team watched exactly this. At 2,000 DAGs their scheduler stopped keeping up, parse times ballooned, and nobody could point at a single broken thing. The defaults that served 50 DAGs were quietly strangling 2,000.

This guide walks the scheduler loop, the three parsing knobs that matter, and the database checks that unstick it. Small numbers. Big relief.

The Scheduler Loop and Where Time Goes

Each scheduler loop does two expensive things. It parses DAG files into the DagBag, then queries the metadata DB to decide which task instances to queue. At 50 DAGs both phases finish in seconds. At 2,000 the parse phase dominates and every loop starts late.

You'll see it first as scheduling drift. Task start times slip minutes per day, new DAGs appear slowly, and the processor log shows total_parse_time climbing. The scheduler isn't broken; it's just doing forty times the work with the same two hands.

Measure before touching knobs. Pull total_parse_time from the processor log, list the five slowest files, and time their imports directly. Most stalls have one hot file doing the damage of hundreds.

Parsing: The Three Knobs That Matter

Three knobs run the parser. parsing_processes sets how many files parse in parallel. min_file_process_interval sets how often an unchanged file gets re-parsed. Together they decide whether the loop finishes in seconds or minutes.

Defaults fit small fleets. One or two parsers with a 30-second interval hum along at 50 DAGs. At 2,000 DAGs that combination serializes thousands of files through a straw while re-parsing unchanged ones twice a minute.

Scale deliberately. Set parsing_processes near 2x scheduler vCPUs minus 1, stretch the interval to 60-120 seconds in prod, and re-measure total_parse_time after one full loop. You'll trade a little change-pickup speed for a lot of scheduling headroom.

Two more dials matter once parsing_processes is sane. dag_dir_list_interval controls how often the folder is rescanned for new files — raise it when DAGs rarely change. file_parsing_sort_mode pushes your freshest edits to the front of the parse queue so deploys show up fast. And keep a tight .airflowignore: every stray file the processor opens is parse budget burned for nothing. In Airflow 3 these knobs live under [dag_processor] (env form AIRFLOW__DAG_PROCESSOR__*), and dag_dir_list_interval is renamed refresh_interval — same job, new address.

Know the timeout pair too. dagbag_import_timeout (default 30s) caps importing one file; dag_file_processor_timeout (default 50s) caps end-to-end processing. A DAG blowing past them shows Broken DAG: Timeout and drops out of scheduling entirely. Raising the timeout treats the symptom — a file needing 90s to import is doing work (DB calls, network fetches) that belongs inside tasks, not at top level.

# scale parsers with scheduler CPUs, stretch the re-parse interval
AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES=7
AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL=90

# verify what is actually live (env beats airflow.cfg)
airflow config get-value dag_processor parsing_processes
airflow config get-value dag_processor min_file_process_interval
nproc  # compare: parsers ~= 2x vCPU minus 1

# watch one full loop after the change
grep total_parse_time $AIRFLOW_HOME/logs/dag_processor_manager/*.log | tail -5

Database Saturation and Index Health

The metadata DB is the scheduler's other half. Every loop scans task_instance and dag_run rows, and at scale those scans dominate DB CPU. You'll spot it in pg_stat_statements before any Airflow metric names it.

Missing indexes are the usual culprit. Composite indexes on the hot filter columns turn full scans into seeks, and the payoff shows up as lower loop latency within minutes. Add them in a maintenance window; index builds lock writes on big tables.

Watch connection pressure too. More parsers and more scheduler threads mean more concurrent DB sessions. Size the pool for the tuned parser count, not the default one, or you'll trade parse stalls for connection waits.

Size the DB link before adding schedulers. Airflow is connection-hungry, and Postgres handles connections per-process, so the standard answer for any non-trivial Postgres install is PgBouncer in front of the metadata DB (the Helm chart ships it). Do the math first: (schedulers + DAG processor + API server) x (sql_alchemy_pool_size + sql_alchemy_max_overflow) must fit inside the DB's max_connections, or you'll trade a parse bottleneck for too-many-connections. Tune sql_alchemy_pool_recycle too, so dead connections get rebuilt instead of wedging loops.

Scale schedulers like adults: Airflow's scheduler scales almost linearly, and two schedulers suit most teams — don't pass three without a measured reason, since each one multiplies DB traffic. In Airflow 3 the standalone DAG processor is mandatory, not opt-in: start a scheduler without a dag-processor and new DAGs silently never appear. Run it as its own service with its own resources so parsing never steals the scheduler's loop.

-- slowest scheduler-facing queries by total time
SELECT query, calls, round(total_exec_time::numeric, 1) AS total_ms
FROM pg_stat_statements
WHERE query LIKE '%task_instance%' OR query LIKE '%dag_run%'
ORDER BY total_exec_time DESC
LIMIT 10;

-- queued-task age per pool: the starvation view
SELECT pool, count(*) AS queued,
       max(now() - queued_dttm) AS oldest_queued
FROM task_instance
WHERE state = 'queued'
GROUP BY pool
ORDER BY oldest_queued DESC;

Monitoring Scheduler Lag Before It Bites

Scheduler lag has one honest metric: heartbeat freshness plus run-creation delay. If heartbeats stay fresh while runs appear late, the parser is behind. If both slip, the whole scheduler host is saturated.

Graph total_parse_time alongside queued-task age. Parse time rising with flat queue age means files are the bottleneck. Both rising means the scheduler can't keep up at all, and parser tuning alone won't save you.

Alert on lag, not just death. A heartbeat-stale page catches dead schedulers; a parse-time trend alert catches the slow drowning weeks earlier. You'll fix stalls on a calm afternoon instead of during month-end.

Tune the scheduler loop itself. max_tis_per_query caps how many task instances one loop examines (keep it at or under parallelism, or query predicates get expensive). max_dagruns_to_create_per_loop (default 10) throttles run creation per tick. scheduler_idle_sleep_time (default 1s) paces empty loops. And remember pool semantics: the scheduler enqueues at most the free slots per pool per iteration, so priority only bites when demand exceeds slots — low-priority tasks can jump ahead inside one batch.

The 2,000-DAG Failure Story

The 2,000-DAG stall had a face: five files with top-level Variable fetches and DB connections. Each import cost seconds, the parse loop serialized behind them, and every DAG in the fleet paid for five teams' shortcuts. Deleting those imports did more than doubling parsers.

The rule is simple to state and hard to enforce. Top-level code builds DAG objects; everything else lives inside task callables with lazy imports. Code review should treat a top-level Variable.get like a production incident waiting to happen.

Enforce it with CI. Time every DAG file import on pull requests and fail the build over a 2-second budget. You'll catch the next 12-second import before it merges instead of after it stalls the fleet.

"""Keep DAG top-level imports pure: heavy work moves into tasks."""
import pendulum
from airflow.sdk import dag, task

@dag(
    dag_id="sales_hourly",
    schedule="@hourly",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    max_active_tasks=4,
    tags=["sales"],
)
def sales_hourly():
    @task
    def extract(partition: str) -> str:
        # lazy import: pandas loads per task run, never at parse time
        import pandas as pd
        from airflow.sdk import Variable

        warehouse = Variable.get("sales_warehouse")
        df = pd.read_parquet(f"{warehouse}/{partition}.parquet")
        return f"rows={len(df)}"

    extract(partition="{{ ds }}")

sales_hourly()

When to Move Scheduling Elsewhere

Some fleets outgrow one scheduler's loop no matter the knobs. Thousands of tiny DAGs with minute-level schedules spend more time parsing than scheduling, and no parser count fixes that ratio. That's a design signal, not a tuning failure.

Consolidate first. Hundreds of single-task DAGs often collapse into mapped tasks inside a few parameterized DAGs. Fewer files means less parsing, and mapped tasks schedule as one unit. You'll cut parse load by an order of magnitude without new infrastructure.

Split schedulers second. Isolate latency-sensitive tier-1 DAGs from bulk backfill fleets so one's parse storm can't drown the other's scheduling. Tuning wins the battle; architecture wins the war.