Airflow Executors: Zero Parallelism by Default? Fix It

The fleet had 100 DAGs and the throughput of one. Tasks queued for hours while CPU graphs stayed flat and engineers blamed the scheduler.

The scheduler was innocent. The executor was Sequential, running one task at a time like a demo laptop. Nobody had changed it since install day.

Executors decide who actually runs your tasks. Pick wrong and no amount of tuning saves you. You'll learn the four options here.

The Executor Job: Who Runs Task Instances

The executor owns one job: turning queued task instances into running processes somewhere. The scheduler decides order; the executor provides the somewhere. Confuse the two and you'll tune the wrong component for weeks. Executor logic runs inside the scheduler process itself, so there is no separate executor daemon to restart; you check it with airflow config get-value core executor.

Sequential runs tasks inline one at a time and exists for dev and tests. Local forks subprocesses on the same host and suits single-node teams. Celery sends tasks through a broker to a fleet of workers. Kubernetes launches a fresh pod per task for full isolation. Remote families split further: queued batch executors (Celery, Batch, Edge) reuse warm workers, while containerized ones (Kubernetes, ECS) spin a clean container per task.

Your executor choice caps everything downstream. No pool or parallelism setting can exceed what the executor physically provides. Since 2.10 you can also mix executors in one deployment (comma-separated list with aliases) and pin tasks via executor='LocalExecutor' or default_args, so steady and spiky work share one control plane.

The Four Executors and Their Ceilings

SequentialExecutor belongs on laptops: zero setup, zero parallelism, SQLite-friendly. LocalExecutor fits teams under roughly 30 DAGs on one sturdy VM: set parallelism to 2x CPU cores and cap per-DAG tasks so one DAG can't flood the box.

CeleryExecutor fits steady fleets with predictable volume. You run a broker plus N workers, scale workers with queue depth, and get second-scale task starts on warm workers. KubernetesExecutor fits bursty or spiky work where tasks need distinct images or resources: each task gets a clean pod, cold start costs 20-60 seconds, and KEDA scales nodes on queue depth.

Choose by shape, not hype. Steady ETL loves Celery's warm workers; nightly ML spikes love Kubernetes' isolation.

# airflow.cfg (single-node team, explicit choice)
[core]
executor = LocalExecutor
parallelism = 32
max_active_tasks_per_dag = 8

# verify live config (env overrides file)
# airflow config list | grep -iE 'executor|parallelism|max_active'

Parallelism vs Max Active Tasks vs Pool

Concurrency applies as a stack and every layer binds. parallelism caps total running tasks instance-wide. max_active_tasks_per_dag caps a single DAG's footprint. Pools cap shared resources like warehouse connections regardless of DAG. Queues route tasks to worker groups with distinct capacity.

Tune bottom-up: pools first to protect external systems, then per-DAG caps to stop one DAG flooding, then global parallelism to match real worker slots. Raising parallelism with a full default_pool changes nothing except your confidence.

Watch the metadata DB as you raise caps. Each running task holds connections and heartbeats; doubling parallelism without DB headroom trades queued tasks for connection timeouts.

Queue Routing That Prevents Starvation

Queues are lane markings. Heavy training tasks on the default queue block quick ETL behind them like a truck in the fast lane. Dedicated queues with matching workers separate traffic by shape.

Tag tasks at authoring time: queue='etl' for warehouse loads, queue='ml' for training, default for lightweight checks. Start workers subscribed to specific queues so capacity follows labels, not hope.

Audit queues quarterly. Labels drift as DAGs evolve, and an ml queue full of ETL tasks is just the default queue with extra steps.

Choosing an Executor: A Scale Table

Pick with a scale table, not vibes. Under 10 DAGs on one box: LocalExecutor, parallelism 16-32. Ten to fifty DAGs with steady volume: Celery with 2-4 workers and a broker. Bursty ML or strict isolation needs: Kubernetes with requests/limits and KEDA. Beyond that, list two executors in [core] executor (e.g. CeleryExecutor,KubernetesExecutor), alias them for clarity, and route per task with executor='...' so short ETL stays warm while training goes isolated. Metrics then publish per executor (executor.open_slots.<name>), so dashboards split by backend.

Factor in team skill. Celery needs broker ops (Redis/RabbitMQ, Flower, visibility timeout). Kubernetes needs cluster ops (Helm, node pools, pod quotas). The fancier executor you can't operate is worse than the simpler one you can.

Revisit yearly. The executor that fit 20 DAGs rarely fits 120. Retired hybrids (CeleryKubernetesExecutor, LocalKubernetesExecutor) are gone in 3.x; the comma-separated multi-executor config replaces them without the queue-field hack.

# celery worker subscribed to its lane
# airflow celery worker -Q etl,default --concurrency 8 --autoscale 12,4

from airflow.sdk import dag, task
from datetime import datetime

@dag(dag_id="nightly_mix", schedule="@daily", catchup=False)
def nightly_mix():
    @task(queue="etl", pool="warehouse_pool")
    def load_warehouse():
        return "heavy load on etl workers"
    @task(queue="default")
    def health_check():
        return "quick check stays unblocked"
    load_warehouse() >> health_check()

nightly_mix()

What Backpressure Looks Like Per Executor

Backpressure looks different per executor. Local shows load average pinned with fork errors. Celery shows broker queue depth climbing while Flower shows idle workers (visibility timeout or queue mismatch). Kubernetes shows pending pods with insufficient CPU or quota errors.

The universal signal is queued-task age: the gap between a task becoming queued and actually running. Ages climbing through peak while capacity looks free mean a cap or route is wrong, not that you need more machines.

Instrument it. Alert when p95 queued age exceeds 10 minutes on prod pools; that page arrives hours before SLA misses do.