Airflow High Availability: The Scheduler That Double-Ran

High availability is the first thing teams ask for and the last thing they configure. Add a second scheduler, call it HA, and go home. Except Airflow schedulers don't sit idle in standby — they all claim work from the same metadata DB, and nothing stops two of them from claiming the same task. That's how our fleet started double-running every DAG.

The honest framing: Airflow HA is not two brains deciding. It's one shared database with several workers that politely coordinate through it. The database is the brain; the schedulers are hands. Lose that framing and every HA setup becomes a double-run machine.

The coordination is real and mostly automatic — leases, heartbeats, DAG processor pools. But "mostly automatic" is where incidents live: the knobs that govern it are config keys most teams never open, and the failure mode is duplicated data, not a crash.

You don't need HA until you do. When you do, you need it boring.

This is how you make it boring.

1. What HA Actually Means for Airflow

High availability for Airflow is not "two schedulers, one leads." Every scheduler you start is active — each one claims DAG runs, queues task instances, and parses DAG files. What keeps that from descending into chaos is the metadata database.

The metadata DB is the coordination point: schedulers write heartbeats, claim leases, and read state from it. The database is the brain; schedulers are hands doing the same job. The architecture is safe because of how the hands coordinate, not because anything stands idle.

So HA has three moving parts: multiple scheduler processes, a shared metadata DB that can handle them, and the coordination knobs that stop them from stepping on each other. Skip any one and you have HA in name only.

The coordination is deliberately database-only: no Raft or Paxos, no ZooKeeper or Consul. Schedulers serialize their claim-and-enqueue step with row-level locks on the Pool table (SELECT ... FOR UPDATE), which is how two schedulers respect the same concurrency limits.

That's the mental model everything else in this article builds on.

2. Multiple Schedulers: Safe With What Caveats

Multiple schedulers are safe — Airflow designed for it — but the safety is conditional. The conditions are version support, a Postgres-grade metadata DB, and coordination knobs set deliberately instead of left at defaults.

The first condition is mundane: only recent Airflow 2.x and 3.x support multiple schedulers properly. On older versions the scheduler was assumed to be a singleton, and the feature is the coordination layer, not the process count. Check your version before you scale.

The second is the database. Schedulers coordinate through it, which means the DB must handle roughly twice the queries and hold locks gracefully. SQLite — fine for a laptop — is a dead end for HA. Postgres with adequate pool size is the floor.

The third condition is where incidents live: ha_interval and related tuning. Leave them at defaults and the coordination is sloppy; set them blind and you break failover. Understand the knob before you turn it.

The docs spell out the database floor precisely: PostgreSQL 12+ and MySQL 8.0+ run any number of schedulers "all ready to go" — no further setup needed — while MariaDB below 10.6 is not supported (it lacks the SKIP LOCKED/NOWAIT clauses) and SQL Server is untested. One lock-related flag matters: use_row_level_locking must stay True — the docs say that with it False, you should not run more than one scheduler at once.

One executor caveat: multi-scheduler HA is built for Celery or Kubernetes executors, where tasks run independently of the scheduler. Running LocalExecutor on each scheduler ties its tasks to its process — restart a scheduler and its tasks die — which quietly defeats the HA you thought you had.

# Coordination knobs on BOTH schedulers — identical values
AIRFLOW__SCHEDULER__HA_INTERVAL=30
AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC=5
AIRFLOW__SCHEDULER__PARSING_PROCESSES=4
AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL=30

# Per-DAG concurrency cap — the duplicate-execution safety net
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG=8

# Verify both schedulers see the same values:
airflow config get-value scheduler ha_interval
airflow config get-value scheduler scheduler_heartbeat_sec

3. The DAG Processor Pool

DAG parsing is the scheduler's most expensive habit, and it's the one that bit us. Every scheduler runs a DAG processor pool that parses the DAG folder into a snapshot the scheduler then uses to create runs and queue tasks.

With one scheduler, the pool is simple. With two, you have two pools parsing the same files into overlapping snapshots — and stale snapshots are where double-scheduling starts. The coordination that prevents this is part of the scheduler's lease and heartbeat machinery, and it works — until versions drift or the folder is large enough that parsing lags.

The practical discipline: one DAG source shared by all schedulers (the same volume or the same git-sync sidecar), identical parsing configuration on every scheduler, and a min_file_process_interval high enough that the pools aren't stampeding the same files every loop.

Parse slowness doesn't look like an HA problem. It looks like stale DAGs, missed runs, and the occasional duplicate. Treat parse health as HA health.

4. Database HA: Read Replicas and Failover

You made the database the brain — so the database itself needs redundancy. Airflow's metadata DB is one writer, and that writer needs a real failover story: managed Postgres with automatic failover, or replication configured so a standby can take over the write role.

Read replicas are tempting for HA, and mostly wrong here. Airflow's schedulers need consistent reads of the state they're about to claim. A read replica with replication lag lets two schedulers both decide they own the same DAG run — the exact double-run we lived through.

The safe pattern: write to one primary, and let Airflow's own retries absorb brief failover windows. If you add replicas for reporting queries, never point scheduler config at them. The scheduler reads what it writes.

Failover for the database is the slowest part of any HA story — plan for minutes, not seconds — and it's the part most runbooks forget to rehearse.

-- Who is claiming what? Duplicate claims are the HA tell.
SELECT dag_id, run_id, task_id, COUNT(*) AS claims
FROM task_instance
GROUP BY 1, 2, 3
HAVING COUNT(*) > 1;

-- Are both schedulers alive and heartbeating?
SELECT process_uuid, last_heartbeat, is_active
FROM scheduler_uptime
ORDER BY last_heartbeat DESC;

-- Replication lag: must stay near zero for scheduler reads
SELECT now() - pg_last_xact_replay_timestamp() AS replica_lag;

5. Detecting Double-Scheduling

The tell for double-scheduling is in the metadata DB before it's in your data warehouse: two task_instance rows for the same dag_run, or a run that claims more task instances than the DAG defines. The grid view shows it as duplicate runs; the SQL shows it as duplicate rows.

But detection shouldn't wait for the duplicate. Two earlier signals precede it: leadership churn and stale heartbeats. If schedulers keep trading the lead, or one scheduler's heartbeat age grows while another shrinks, the coordination is failing before the tasks are.

Then there's the operational signal: task age. When two schedulers queue the same work, the queue behaves oddly — tasks complete in bursts and the queue depth never matches the runbook. Alert on the difference between DAG runs triggered and runs completed.

Detection is a query you can run today, plus two metrics you should alert on tomorrow. Build both before you need either.

What happens next is adoption, not restart: when a scheduler is detected dead, its running and queued tasks are "adopted" by the survivors — the sweep is governed by orphaned_tasks_check_interval and the liveness threshold scheduler_health_check_threshold. And the CLI ships an HA-ready liveness check: airflow jobs check --job-type SchedulerJob --allow-multiple --limit 100 succeeds as long as at least one scheduler is alive.

# 1. Duplicate task instances — the smoking gun
airflow tasks states market_etl load_orders 2026-07-15

# 2. Runs claiming more tasks than the DAG has
SELECT dag_id, run_id, COUNT(*) AS task_count
FROM task_instance
GROUP BY 1, 2
HAVING COUNT(*) > 6;  -- our DAG defines 6 tasks

# 3. Heartbeat age per scheduler — leader vs laggard
SELECT process_uuid, now() - last_heartbeat AS age
FROM scheduler_uptime;

# 4. The config knob that governs claim timing
airflow config get-value scheduler ha_interval

6. Health Endpoints and Liveness

A process can be alive and useless. The scheduler health endpoint answers the two questions that matter: is the scheduler process up, and can it reach the metadata database? Both must return healthy, or your HA setup is a demo.

Liveness for the fleet means polling those endpoints on a schedule and alerting on the answer, not on process exit codes. A scheduler that lost its DB connection exits nothing — it hangs. The endpoint catches it; the process monitor doesn't.

The metrics that complete the picture are scheduler_heartbeat freshness and leadership stability. Prometheus, CloudWatch, or a cron curl all work — the mechanism matters less than the fact that a human gets paged before the queue does.

Rehearse the failover too: kill scheduler-1 in staging and measure how long scheduler-2 takes over. The first time you run that test should be in staging, not in the middle of the night.

The declaration of death is configurable too: scheduler_health_check_threshold sets how stale a heartbeat must be before a scheduler is declared dead and its tasks become adoptable — tune it against your DB latency so a slow database doesn't trigger phantom failovers.

# The two answers that matter
curl -s http://localhost:8793/health
# {"metadatabase": "healthy", "scheduler": "healthy"}

# Rehearsed failover, in staging:
# 1. kill scheduler-1
# 2. watch scheduler-2 claim leadership
watch -n 5 'curl -s http://scheduler-2:8793/health'

# 3. prove runs still execute (and only once)
airflow dags list-runs market_etl --state running

7. Incident Deep Dive: Failure Flow → Fix

Replay our night. We scaled from one scheduler to two for a planned HA rollout. Both schedulers parsed the same DAG folder, wrote heartbeats to the same Postgres, and — with ha_interval at default and no coordination review — both claimed the same DAG runs. The grid view filled with duplicate runs; the warehouse loaded every partition twice.

The failure flow was textbook: overlapping DAG parsing produced stale snapshots, the lease machinery accepted both claims, and nothing at the concurrency layer capped the duplicate execution. The database happily served two healthy clients asking for the same work. The first scream came from the data team at 06:00.

The fix had three layers. First, coordination: ha_interval tuned deliberately, identical env blocks on both schedulers, one shared DAG source. Second, defense in depth: max_active_tasks_per_dag capped so a duplicate claim could never fully double-execute. Third, detection: scheduler_heartbeat metrics wired to alerting, plus the duplicate-claim query in the runbook.

HA never double-runs a healthy setup by accident. It double-runs a sloppy one — now ours is the former.