Airflow Performance: Scheduler Stalls at 2,000 DAGs

The scheduler does three jobs in a loop: parse DAG files, create runs for what's due, and hand tasks to the executor. For most teams, step one eats 90% of the time. That's the whole performance story.

At 2,000 DAG files our scheduler stopped scheduling. Not crashed — stopped. The UI was fine, the database was fine, but the parse loop took so long that runs never got created. Every tuning knob we'd ignored for a year was suddenly the only thing that mattered.

We'll walk the parse loop, the three knobs that control it, the database side, and how to monitor scheduler lag. It ends with the honest question: when do you stop tuning and redesign?

Parsing isn't the price of using Airflow. It's the thing you manage.

1. The Scheduler's Loop and Where Time Goes

The scheduler loops forever: pick up DAG files, parse them into a dagbag, decide which runs are due, and hand tasks to the executor. Parse is the step that scales with your codebase.

Each DAG file is imported as Python. The import runs the whole module — including anything at top level. A clean file parses in 100-300ms; a file that opens a DB connection at import time takes seconds.

That difference compounds. With 2,000 files, one bad pattern in fifty files adds minutes to every loop.

Measure the folder before you touch a single knob.

Two loop constants worth knowing: each loop creates at most max_dagruns_to_create_per_loop DagRuns (default 10), and scheduler_idle_sleep_time (default 1s) paces the loop when there's nothing to do.

import time
from airflow.models.dagbag import DagBag

start = time.perf_counter()
bag = DagBag(dag_folder="/opt/airflow/dags", include_examples=False)

elapsed = time.perf_counter() - start
print(f"parsed {len(bag.dags)} DAGs in {elapsed:.2f}s")
for dag_id, dag in bag.dags.items():
    print(f"{dag_id}: {len(dag.tasks)} tasks")

2. Parsing: The Three Knobs

Knob one: parsing_processes (max_threads in older Airflow 2.x) — how many processes parse DAG files in parallel. Default is 2. At 2,000 DAGs, that's 1,000 files per process per loop.

Knob two: min_file_process_interval — the minimum time between parsing attempts of the same file. Default 30s; too low thrashes the loop, too high delays new DAGs.

Knob three: dagbag caching. Airflow 3 caches parsed DAGs across loops so unchanged files skip re-import — the single biggest win at scale.

Raise processes to match cores, keep the interval at 30-60s, and let the cache do the heavy lifting.

The real ceiling on parsing_processes is memory, not CPU: each process is a full Python interpreter, and while Airflow uses forking and copy-on-write to share it, any import after the fork breaks the sharing and memory climbs. In Airflow 3 both knobs moved to the [dag_processor] section — AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES — and min_file_process_interval still exists there, default 30s.

export AIRFLOW__SCHEDULER__PARSING_PROCESSES=10
export AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL=60
export AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT=60

3. Database Saturation and Index Health

The scheduler reads the metadata DB on every loop: which runs exist, which tasks are due, which heartbeats are fresh. When the DB slows, the scheduler slows — even with perfect parse times.

Task instance tables grow without bound unless old runs are cleaned. Bloat means dead rows, bigger scans, and index pages that miss the cache.

The fix is boring: index the hot tables, clean old runs, and watch the slowest queries.

SELECT relname, n_live_tup, n_dead_tup, last_vacuum
FROM pg_stat_user_tables
WHERE relname IN ('task_instance', 'dag_run', 'xcom')
ORDER BY n_dead_tup DESC;

4. Monitoring Scheduler Lag

Lag is the gap between now and the last completed parse cycle. Airflow exposes it as dag_processing.last_run.seconds_ago; Prometheus makes it alertable.

Lag under 60 seconds is healthy. Above 120 seconds, runs start firing late. Above 300, you're in the stall zone — runs may never be created at all.

Every tuning change you make should be validated against this single number. Before, during, and after.

groups:
  - name: airflow_scheduler
    rules:
      - alert: SchedulerLagHigh
        expr: max(time() - airflow_dag_processing_last_run_seconds_ago{})
          > 120
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Scheduler parse lag over 120s"

5. The 2,000-DAG Failure Story

Month one: 400 DAGs, one parse process, parse times under 200ms each. Nobody checked; nothing broke.

Month nine: 2,000 DAGs. Four legacy files still opened DB connections at import time — each added 8-12 seconds. With a single processor, one loop took 40+ minutes. Runs were due every 5 minutes.

The scheduler was drowning and the queue looked like a backlog, so the first instinct was to raise task parallelism. It changed nothing; the bottleneck was upstream of the queue.

The actual fix took an afternoon: processes to 10, the four files cleaned, caching enabled. Lag dropped from 2,400 seconds to 12.

2,400 to 12. Same hardware.

6. When to Move Scheduling Elsewhere

Tuning has a ceiling. Beyond it, the honest answer is redesign: fewer DAG files via config-driven generation, or fewer DAGs per platform.

If your folder has 5,000 near-identical DAGs, generated-from-config DAGs replace thousands of files with dozens. If a team ships 50 small DAGs a day, a second Airflow instance per domain beats one giant shared one.

And if DAGs are many but tasks are few — hundreds of one-task DAGs — you don't have an Airflow problem. You have an architecture problem.

Know which phase you're in before you spend another hour tuning.

Two scale-out moves change the shape instead of the settings. A standalone DAG processor separates parsing from scheduling — in 2.x opt in with AIRFLOW__SCHEDULER__STANDALONE_DAG_PROCESSOR=True and run airflow dag-processor; in Airflow 3 it's a first-class split that scales independently and keeps the scheduler from executing author code. And the scheduler scales almost linearly, so a second scheduler is a real multiplier — but every additional scheduler multiplies metadata-DB connection traffic, which is why PgBouncer in front of Postgres is the standard companion.

7. A Tuning Checklist for Production

Tune in this order and stop when lag is under 60 seconds.

1. Profile the folder — find files parsing over 1 second.
2. Fix slow files — top-level DB calls are the usual culprit.
3. Raise parsing_processes to match cores.
4. Keep min_file_process_interval at 30-60s; rely on caching.
5. Check DB bloat and prune old runs.
6. Alert on lag and re-run the checklist after every deploy.

And once the folder passes ~1,000 files, set file_parsing_sort_mode to modified_time so recently changed DAGs parse before the long tail of stable ones.

Boring, measurable, reversible. That's what production tuning looks like.