Airflow High Availability: Scheduler That Double-Ran

High availability doubled every DAG instead of protecting them. Two schedulers meant two copies of each run writing the same tables.

Redundancy without coordination is just concurrency with confidence. The second scheduler did exactly what the first did, at the same time.

HA in Airflow is a tuned pair, not a spare. You'll learn that tuning here.

What HA Actually Means for Airflow

HA in Airflow covers four things: schedulers that survive crashes, a database that survives failover, webservers behind a load balancer, and parsing that keeps up with DAG count. Miss any one and the other three can't compensate. Astronomer's resilience checklist names the same set: redundant schedulers and webservers, replicated Postgres or MySQL, distributed executors, and idempotent DAGs.

Two schedulers provide active-active redundancy: both parse and queue, coordinated through the metadata DB with no Raft or ZooKeeper on the side. HA Postgres provides one writable primary with standbys. Processor pools provide parse throughput per scheduler, and extra webservers behind HAProxy or an ALB keep the UI up when one fails.

Don't call one scheduler plus hope HA. It's a single point of failure with a monitoring dashboard.

Multiple Schedulers: Safe With Caveats

Multiple schedulers are safe when DAGs are idempotent and capped. Each scheduler parses independently and claims work through the DB; races resolve correctly when reruns are no-ops. Postgres 12+ and MySQL 8.0+ are ready out of the box with zero extra flags: just start more scheduler copies. MariaDB needs 10.6+ for SKIP LOCKED/NOWAIT support, and SQL Server HA remains untested, so don't stake prod on either.

The safety hinge is row-level locking. The critical section takes write locks across the Pool table (SELECT ... FOR UPDATE) so limits hold while task instances move to queued. Flip use_row_level_locking to False and you must run exactly one scheduler. Tune heartbeat intervals so failover detects death in tens of seconds without flapping. Ten-second heartbeats with a 30-second health threshold catch crashes fast while surviving GC pauses.

Verify with duplicate audits after enabling the pair. Two weeks of zero duplicate logical dates proves the tuning; assumptions prove nothing.

[scheduler]
# two schedulers, coordinated heartbeats
scheduler_heartbeat_sec = 10
scheduler_health_check_threshold = 30
max_threads = 4

[dag_processor]
parsing_processes = 4
dag_dir_list_interval = 60

# run two scheduler replicas (compose or k8s)
# docker compose up -d --scale airflow-scheduler=2
# airflow dags list-jobs -o table  # both heartbeats must stay fresh

The DAG Processor Pool

The DAG processor pool parses files into the DB so schedulers queue from metadata, not files. parsing_processes sets parallel parsers; dag_dir_list_interval sets rescan cadence. Under-provisioned pools lag identically on every scheduler. The scheduling loop itself batches: max_dagruns_to_create_per_loop locks how many DAGs one scheduler claims for run creation (lower it for 10k-task monsters), max_dagruns_per_loop_to_schedule bounds examined runs per pass, and max_tis_per_query caps task rows per query at or below core.parallelism.

Slow imports hurt pools most: a 12-second top-level DB connection times parser count stalls the whole fleet. Keep top-level code pure and lazy-load clients inside tasks. Orphaned-task checks (orphaned_tasks_check_interval) plus dead-scheduler adoption mean a crashed scheduler's running tasks get supervised by survivors instead of drifting.

Scale pools with DAG count, not scheduler count. Doubling schedulers over a slow file set doubles cost while lag stands still. When the scheduler process is CPU-bound, extra schedulers scale nearly linearly until the shared DB becomes the ceiling, so watch connection counts and consider PgBouncer on Postgres fleets.

Database HA: Replicas and Failover

Schedulers need one writable primary; the UI and audits can read from replicas. Managed Postgres with automated failover plus connection pooling handles the pattern without heroics. Postgres fleets of any size should front the DB with PgBouncer: Airflow is connection-hungry and process-based Postgres accounting runs out long before MySQL-style thread pools do. The Helm chart ships PgBouncer out of the box.

During failover, schedulers pause writes and resume on the new primary. Tasks already running continue; new scheduling waits seconds, not hours. Test failover quarterly so the runbook works when it matters.

Never multi-master the metadata DB. Split-brain writes corrupt run state in ways no backfill can repair. Idle-loop pacing via scheduler_idle_sleep_time keeps quiet schedulers from hot-spinning while letting busy ones chain loops back-to-back.

Detecting Double-Scheduling

Duplicates announce themselves as identical logical dates with minute-apart starts. Audit weekly with list-runs output piped through a counter. Any duplicate on a non-idempotent DAG is a data incident, not a curiosity.

Prevent with caps and idempotency: max_active_runs bounds concurrency per DAG, partition-keyed writes absorb races. Alert on duplicate creation so the next race pages within minutes.

Keep the audit after tuning. Parser timing drifts as DAGs grow, and today's safe overlap becomes next quarter's race.

# duplicate-run audit (run weekly after enabling HA)
airflow dags list-runs sales_daily -o json > /tmp/runs.json
python3 -c "
import json, collections
runs = json.load(open('/tmp/runs.json'))
keys = [r.get('logical_date') or r.get('run_id') for r in runs]
dups = [k for k, c in collections.Counter(keys).items() if c > 1]
print('duplicate logical dates:', dups if dups else 'none')
"
# heartbeat freshness
# airflow dags list-jobs -o table  # scheduler heartbeats < 60s

Health Endpoints and Liveness

Liveness probes should hit scheduler health endpoints and restart dead instances automatically. In Kubernetes that's a livenessProbe on the scheduler deployment; on compose it's a healthcheck plus restart policy.

Expose scheduler_heartbeat age, queued-task age, and parse duration to your metrics system. Page when any scheduler exceeds 60 seconds stale or queue age passes 10 minutes.

Document the runbook: which scheduler died, how to restart, how to verify re-election. Dead schedulers at 3 AM shouldn't require design meetings.