Airflow Task States: Stuck in Queued? Here's the Fix

Every Airflow UI shows the same color-coded lie: green tasks are fine, red tasks are not. The truth lives in the states in between. Queued is where pipelines quietly die — tasks waiting for a worker slot that never comes, while the DAG run hangs for hours. Failed with no retry is where pipelines die loudly.

Our nightly pipeline failed on a single transient database blip. The task had retries=0 and no backoff, so one 500 from Postgres turned a 20-minute run into a full failure. The fix wasn't more compute. It was a retry policy and an understanding of what queued actually means.

This article maps the full task state machine, then shows you the retry and concurrency knobs that keep it healthy. States aren't trivia. They're the diagnostics that tell you whether your pipeline is slow, stuck, or actually broken.

1. The Full State Machine (and Why It Matters)

A task instance starts life in queued — the scheduler has decided to run it, but a worker hasn't picked it up yet. From there it can go running, then success, failed, or skipped. Failed hands the baton to up_for_retry when retries remain. Up for reschedule and deferred are the waiting states for sensors and deferrable operators.

Every state is a diagnostic. Queued says "no capacity." Up for retry says "a transient failure happened, and we're waiting to try again." Skipped says "this task's trigger rule decided it had nothing to do." A state is never just a color.

Read the grid view like a doctor reads vitals: a run full of green with one yellow up_for_retry is healthy; a run full of queued is a capacity problem wearing a scheduler's costume.

One refinement from the docs' canonical flow: a fresh task instance actually starts in none (dependencies not yet met), moves to scheduled when the scheduler determines it should run, and only then to queued, when the executor accepts the task and it awaits a worker. Three more states complete the picture: restarting (a running task cleared or restarted externally), upstream_failed (an upstream task failed and the trigger rule requires it), and removed (the task vanished from the DAG after the run started).

2. Retries: Honest Retry vs Hiding Bugs

Retries exist for transient failures: network blips, connection timeouts, database 500s, race conditions with an upstream job. They exist to absorb noise, not to launder bugs. A deterministic failure — a typo in SQL, a missing column — will fail the same way on retry 1 and retry 10.

The line between honest retry and hiding bugs is drawn by your logs. If three retries all fail at the same line of code, you're paying for a bug, not a blip. Check the logs after the first retry, not after the third.

Our incident pipeline had retries=0, which made every transient failure fatal. The fix was retries=3 — enough to absorb a blip, few enough that a real bug surfaces in under an hour.

Airflow also ships the honest-retry escape hatch: raise AirflowFailException inside a task to mark it failed and skip any remaining retries, or AirflowSkipException to skip it outright — the right call when the failure is definitively not transient, like a bad input record. Airflow 3's retry policies go further, letting you attach per-exception retry behavior that returns RETRY, FAIL, or DEFAULT actions from the task worker process without touching task code.

from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator

with DAG(
    dag_id="daily_revenue_rollup",
    schedule="@daily",
    start_date=datetime(2026, 8, 1),
    catchup=False,
    default_args={
        "retries": 3,
        "retry_delay": timedelta(minutes=5),
    },
    tags=["billing"],
) as dag:
    def aggregate_revenue():
        # transient DB blips get retried; real bugs surface fast
        ...

    aggregate_task = PythonOperator(
        task_id="aggregate_revenue",
        python_callable=aggregate_revenue,
    )

3. Backoff and max_retry_delay

Retry count without delay means a hammering thundering herd: three tasks all failing on the same downstream outage, retrying in lockstep every 30 seconds. Backoff is what spreads the attempts out. Airflow's exponential backoff doubles the wait each round: 5 minutes, then 10, then 20.

max_retry_delay is the ceiling. It caps the doubling so a long chain of retries can't stretch into a multi-hour wait. Set it to the maximum time you're willing to let a task sit in up_for_retry before it either succeeds or fails for good.

A sane production default: retries=3, retry_delay=5 minutes, exponential backoff, max_retry_delay=1 hour. That's a worst case of about 35 minutes of waiting for a task that genuinely needs the world to calm down.

from datetime import timedelta

with DAG(
    dag_id="daily_revenue_rollup",
    schedule="@daily",
    start_date=datetime(2026, 8, 1),
    catchup=False,
    default_args={
        "retries": 3,
        "retry_delay": timedelta(minutes=5),
        "retry_exponential_backoff": True,
        "max_retry_delay": timedelta(hours=1),
    },
) as dag:
    ...

4. What "Queued" Means — and Why Tasks Get Stuck

Queued is the waiting room. The scheduler has committed to the run, but no worker slot is free. With the default SequentialExecutor that's one slot total. With Local or Celery executors, slots are bounded by parallelism, pools, and max_active_tasks.

Tasks get stuck in queued for exactly four reasons: no free worker slots (parallelism saturated), a pool with zero free slots, a max_active_tasks ceiling on the DAG, or a scheduler that stopped heartbeating. Diagnose in that order and you'll find it every time.

Our queued pile-up was capacity, not breakage: the DAG's tasks were all competing for a pool sized for half the workload, so late runs queued behind earlier ones and never recovered. The fix was concurrency tuning, not code changes.

One gotcha the docs and scheduler code make explicit: a task stuck in queued is not parked forever. The scheduler eventually fails tasks that linger in queued too long, and that failure counts against the task's retries — so a capacity bottleneck quietly eats your retry budget while the run looks merely 'slow'.

[core]
parallelism = 32            # total task instances across the whole deployment
max_active_tasks_per_dag = 16
max_active_runs_per_dag = 4  # DAG runs per DAG that can be active at once

[scheduler]
max_tis_per_query = 16       # task instances the scheduler schedules per loop

5. Per-DAG vs Per-Task Concurrency Knobs

Concurrency in Airflow is a stack, not a single switch. At the top, [core] parallelism caps total running task instances for the whole deployment. Below it, max_active_tasks_per_dag caps each DAG, max_active_runs_per_dag caps concurrent runs of the same DAG, and pools cap specific task groups regardless of DAG.

The confusion comes when people set one knob expecting it to do another's job. Raising parallelism won't help if a pool is the bottleneck. Raising the pool won't help if the DAG's max_active_tasks is the ceiling.

The rule: reason top-down. Slots exhausted → parallelism. One DAG hogging everything → max_active_tasks_per_dag. A specific external system being hammered → a pool with a priority_weight policy.

from airflow import DAG
from airflow.operators.python import PythonOperator

# Cap API-heavy tasks across all DAGs at 4 concurrent instances
with DAG(
    dag_id="daily_revenue_rollup",
    schedule="@daily",
    start_date=datetime(2026, 8, 1),
    catchup=False,
    default_args={"pool": "external_api_quota"},
) as dag:
    def call_external_api():
        ...

    api_task = PythonOperator(
        task_id="call_external_api",
        python_callable=call_external_api,
    )
# create the pool: airflow pools set external_api_quota 4 "API calls quota"

6. Trigger Rules: The Quiet Killer (Intro)

A task's trigger rule decides when it may start after its upstream tasks finish. The default is all_success — every upstream task must succeed. That's correct for linear pipelines and quietly fatal for anything with branches or skips.

Skipped tasks fail all_success joins. A sensor that soft-fails, a branch you didn't take, a downstream task of either — all of them make the join fail even though nothing actually broke. That's next article's deep dive, but the intro belongs here: trigger rules are states' second half.

The state machine doesn't end at the task. It ends at the DAG's trigger rules deciding whether the graph continues. When a run fails with green upstream tasks, look at the trigger rule, not the code.

7. Monitoring Task States in Production

States only help if someone is watching. The metric that matters most is queued-task age — how long tasks wait for a slot. A rising queue with healthy scheduler heartbeats is a capacity alert. A queue that grows while heartbeats stop is a dead scheduler alert. Two very different on-call pages from one metric.

Failure rate is the second metric, but count it honestly: failed-with-retry-remaining is noise, failed-final is signal. Airflow's own metrics and the task instance list in the UI give you both, and exporting them to Prometheus turns states into trends.

The discipline that ends incidents: a weekly check of the grid view, an alert on queued age, and a retry policy that's visible in the DAG code. Monitoring isn't a dashboard. It's a habit.