Airflow Executors: Zero Parallelism by Default? Fix It

Every Airflow deployment has one: a scheduler that schedules, and an executor that does the actual work. Most teams never touch the second part, and that's the whole problem.

The default executor is SequentialExecutor. It runs every task instance one at a time, in one process, forever. Your 100 DAGs will parse, your tasks will show "queued" then "running" in the grid view, and everything will look alive. Nothing will be parallel.

Executor choice is a capacity decision, not a config footnote. It decides how many tasks run at once, where they run, and how your fleet fails under load. Get it wrong and you'll be explaining queued backlogs at 2 AM.

Pick the executor for the scale you actually have, then set the concurrency knobs deliberately. The defaults are designed for tutorials, not teams.

1. The Executor Job: Who Actually Runs Task Instances

The scheduler is the brain, but it never executes anything. When a DAG run fires, the scheduler turns each task into a TaskInstance and hands it to the executor. The executor is the arm: it finds capacity and runs the task's operator code.

The metadata DB holds the contract between them. The scheduler writes task instances to the DB; the executor picks them up, marks them running, and reports the outcome. That's why executors can be swapped without touching your DAGs.

Four executors ship with Airflow: SequentialExecutor, LocalExecutor, CeleryExecutor, and KubernetesExecutor. They differ in one question: where does the task process run, and how many can run at once?

Ask that question before picking one. It answers itself at scale.

Architecture note from the docs: the executor is a configuration property of the scheduler and runs inside the scheduler process — which is why changing it is a scheduler restart, never a DAG change.

2. The Four Executors and Their Ceilings

SequentialExecutor runs one task at a time inside the scheduler process. It exists so you can start Airflow with zero setup. Its ceiling is one task, period.

LocalExecutor spawns separate processes on the scheduler host, bounded by parallelism. It's the right jump from Sequential: real concurrency, no new infrastructure.

CeleryExecutor ships tasks to a Celery worker fleet over a broker. Workers can live anywhere and scale horizontally, each with its own concurrency cap.

KubernetesExecutor creates one pod per task instance. Full isolation, per-task resources, but pod cold starts add seconds to every run.

Set the executor in airflow.cfg, then restart the scheduler. Nothing else changes.

One version check changes the story of this article's incident. "SequentialExecutor by default" is the Airflow 2.x behavior: the 2.x docs configure it by default and warn that some features, sensors for example, don't work properly under it. Airflow 3.x ships LocalExecutor as the configured default, with [core] parallelism set to 32. The tutorial-flavored default changes between majors, so verify what you actually run with airflow config get-value core executor before trusting the grid view.

LocalExecutor has two container gotchas straight from the docs: its worker processes are subprocesses of the scheduler, so a busy parallelism can read as scheduler memory and trigger OOM restarts — size parallelism to the container limit; and the old unlimited parallelism=0 option was removed in Airflow 3.0.0, the value must be greater than zero.

[core]
executor = LocalExecutor
parallelism = 8

[scheduler]
max_active_tasks_per_dag = 4

[celery]
# used only when executor = CeleryExecutor
broker_url = redis://redis:6379/0
result_backend = db+postgresql://airflow:airflow@postgres:5432/airflow

3. The Concurrency Stack: parallelism, max_active_tasks_per_dag, Pools

Three knobs control concurrency, and they stack. parallelism is the global cap: how many task instances the scheduler will run at once across the whole deployment.

max_active_tasks_per_dag caps how many tasks of one DAG run simultaneously. A fan-out of 50 mapped tasks can still respect a cap of 6.

Pools cap concurrency on a logical resource: the database writer, the API rate limit, the GPU queue. Tasks declare pool="db_writer" and wait their turn regardless of executor.

The effective ceiling is the smallest gate you hit. The graph view shows states; the queue shows pressure. When tasks wait, find which gate they're stuck behind.

Set all three deliberately. Defaults hide contention until it bites.

from airflow.decorators import dag, task
from datetime import datetime

@dag(
    dag_id="market_etl",
    schedule="@daily",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    max_active_tasks=6,
    max_active_runs=1,
    tags=["finance", "etl"],
)
def market_etl():
    @task
    def extract_stock_data(symbol: str) -> str:
        return f"extracted:{symbol}"

    @task
    def load_to_warehouse(payload: str) -> str:
        return f"loaded:{payload}"

    for symbol in ["AAPL", "MSFT", "NVDA"]:
        load_to_warehouse(extract_stock_data(symbol))

market_etl()

4. Queue Routing: Which Worker Takes This Task

Once an executor can scale, you need a way to send work to the right place. That's the queue attribute on a task.

A Celery worker starts with --queues=default,heavy_ml. A task with queue="heavy_ml" only runs on workers listening for it. Same idea on KubernetesExecutor: task queues map to pod configurations, so a GPU job can schedule onto GPU-capable pods.

This is how teams separate fast batch from long ML runs without copying DAGs. It's also the quietest failure mode in Airflow: a task queued to a queue nobody listens on waits forever and nobody gets paged.

Audit your queues when tasks idle. The mapping is the contract.

from airflow.operators.python import PythonOperator


def train_forecast_model():
    # GPU-bound training: only runs on heavy_ml workers
    pass


train_model = PythonOperator(
    task_id="train_forecast_model",
    python_callable=train_forecast_model,
    queue="heavy_ml",
    dag=dag,
)

5. Choosing an Executor: A Scale Table

Here's the honest scale table. One machine, one dev: SequentialExecutor is fine and even convenient. A single machine running a real daily batch: LocalExecutor with parallelism tuned to your cores.

Two or more machines, or steady nightly volume: CeleryExecutor with a Redis or RabbitMQ broker and workers sized by memory. Teams that already run Kubernetes, or need per-task isolation and clean scaling: KubernetesExecutor.

Mixed workloads are normal. You can run CeleryExecutor for the steady batch and route heavy tasks to Kubernetes pods. Both under one scheduler, one metadata DB.

The wrong pick costs you either idle capacity or constant queue pressure. Revisit it when your task volume changes by 10x, not when it doubles.

Since Airflow 2.10.0 there's a first-class way to mix executors without queue tricks: configure several comma-separated in [core] executor and assign tasks with the executor parameter (or per DAG through default_args). The first entry is the environment default; name an executor that isn't in the config and the DAG fails to parse with a UI warning. The old statically-coded hybrids (LocalKubernetesExecutor, CeleryKubernetesExecutor) were removed in Airflow 3.0.0 — the multi-executor config is their replacement.

6. Backpressure: What Each Executor Looks Like When It Fails

Every executor has a signature failure. Sequential and Local fail as host exhaustion: memory pressure, slow scheduler, tasks queuing behind each other with no obvious culprit.

Celery fails as a silent backlog: workers dead, broker alive, tasks queued for hours with zero alerting. The UI shows queued, the calendar fills with late runs.

Kubernetes fails as eviction chaos: no resource limits, pods get OOM-killed, kubelet evicts them, and tasks fail at startup or crash mid-run.

Backpressure looks the same everywhere, though: queued tasks with age. Watch queued-task age, not the grid view. It's the only signal all four executors share.

Set an alert on it before your customers do.