Airflow Celery: Dead Workers Nobody Noticed? Fix It

CeleryExecutor is how Airflow grows from one machine to a fleet. The scheduler stays put; the work travels over a broker to workers that can live anywhere.

Workers are stateless and replaceable, which makes them great. It also makes them disposable: when a worker OOMs at 3 AM, nothing brings it back. Celery doesn't restart workers. It doesn't page anyone. It just lets the queue grow.

That's the whole story of this article: the difference between a platform that looks healthy and one that is healthy. Flower, heartbeats, queue depth, and a visibility timeout are the difference.

Treat your workers as cattle, and build the fences yourself.

1. Why Celery: Decoupled Task Execution

LocalExecutor runs tasks on the scheduler host. That couples scheduling to execution: the moment your nightly batch needs more cores, you're buying a bigger scheduler.

Celery decouples them. The scheduler hands task instances to a broker, and any number of workers pull from the broker wherever they live. Capacity becomes a fleet problem, not a host problem.

Workers report results back to the metadata DB through a result backend. The scheduler never talks to workers directly: broker in, result backend out, contract in between.

That contract is why workers are disposable. Any worker can run any task. The fleet just needs to exist.

One consequence worth internalizing: with CeleryExecutor the scheduler never runs tasks itself — it only queues them. Any node that must also execute work needs its own worker process running there.

2. Broker Setup: Redis vs RabbitMQ

The broker is the queue where task instances wait for a worker. Two brokers dominate: Redis and RabbitMQ.

Redis is the pragmatic pick: fast, already in your stack, one container away. Its queue mechanics are simple, and most Airflow installs never need more. RabbitMQ is the heavyweight: real queue semantics, per-queue persistence, and finer control over delivery.

The visibility-timeout behavior differs. On Redis, unacked tasks are re-delivered after a configurable visibility timeout; on RabbitMQ, the ack timeout governs redelivery. Both can double-execute a task if misconfigured.

Pick Redis for speed and simplicity, RabbitMQ when you need robust queueing semantics. Either works; neither forgives a missing timeout.

One config note: Airflow's visibility_timeout is only honored for Redis and SQS brokers; RabbitMQ has no such knob — its redelivery is governed by Celery's ack semantics instead.

# Redis broker: one container, zero config
docker run -d --name airflow-redis -p 6379:6379 redis:7-alpine

# RabbitMQ broker: heavier, richer queue semantics
docker run -d --name airflow-rabbitmq -p 5672:5672 -p 15672:15672 \
  rabbitmq:3-management

# Point Airflow at the broker
airflow config get-value celery broker_url

3. Workers, Queues, and Autoscale

A worker runs with flags that define its identity: -H gives it a name, --queues lists what it listens on, and --autoscale sets its concurrency window. The window is the memory contract: 12,3 means up to 12 concurrent tasks when the queue is deep, settling to 3 when it's quiet.

Queues route work. A worker started with --queues=default,heavy_ml only takes tasks queued to those names. Fast batch and long GPU jobs can share a fleet without starving each other.

Autoscale is elasticity, not survival. It grows concurrency within one worker process; it cannot resurrect a dead host. A fleet of zero workers autoscales to zero.

Size the concurrency window to memory. Autoscale above your RAM and you're scheduling OOM kills.

# Worker on host prod-1, listening on default and heavy_ml queues
# Autoscale: up to 12 concurrent tasks, settling to 3 when idle
airflow celery worker -H worker@prod-1 \
  --queues=default,heavy_ml \
  --autoscale=12,3 --loglevel=INFO

# Watch it start
airflow celery worker --help | grep autoscale

4. Flower: Fleet Visibility in One Tab

Flower is the Celery web UI. One command, one port, and you can see every worker, its heartbeat, the tasks it's running, and the queues in front of it. Start it with airflow celery flower --broker=redis://redis:6379/0 and open port 5555.

This is the screen that would have saved the night: worker count, heartbeat freshness, and queue depth at a glance. Zero workers with stale heartbeats is the failure signature of a dead fleet.

Flower alone isn't alerting, though. A dashboard nobody watches is decoration. Export the numbers: worker count and queue depth to Prometheus, alerts to Slack or PagerDuty.

Every fleet needs both: the tab to look at, and the alert that looks for you.

One dependency to know: Flower relies on Celery's remote control (worker_enable_remote_control). Disabling that setting to suppress the .*reply-celery-pidbox reply queues Celery creates on unsupported brokers also breaks Flower — the two are tied together.

5. Worker Death: Detect, Restart, Drain

Celery's failure mode is silence. A worker dies, tasks stay queued, and nothing happens. The fix is supervision: a process manager that treats workers as disposable.

systemd is the classic answer on VMs: Restart=always, RestartSec=10, and a worker that dies gets reborn within seconds. Kubernetes plays the same role for containerized fleets.

Draining matters when you take a worker down on purpose. Stop accepting new tasks, let the running ones finish, then stop the process. A kill while tasks run mid-flight is how double execution starts.

The pattern is always the same: detect via heartbeat, restart via supervisor, drain before maintenance.

[Unit]
Description=Airflow Celery Worker
After=network-online.target

[Service]
User=airflow
ExecStart=/opt/airflow/airflow celery worker \
  -H worker@prod-1 --queues=default,heavy_ml --autoscale=12,3
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

6. The Visibility-Timeout Double-Execution Trap

Here's the trap that makes task logs lie. On Redis, a task is a message. When a worker fetches it, the message enters an unacked state with a window: the visibility timeout.

If the worker dies or takes longer than the window, Redis considers the task lost and re-delivers it. Another worker runs it. Your task executed twice, and both runs wrote "success."

Airflow's default visibility timeout for Redis and SQS brokers is 86400 seconds — 24 hours — so the classic failure is a task that runs longer than a day: the broker redelivers it while the original worker keeps going, and both complete. The UI and logs show the tell: 'Task Instance Not Running' FAILED: Task is in the running state. Set visibility_timeout above your longest realistic task runtime and re-check it whenever a new long task ships. Airflow also sets task_acks_late=True by default, but on Redis and SQS that does not override the visibility timeout — the broker still redelivers once the window expires.

This is also why tasks must be idempotent: even with correct timeouts, crashes and redelivery happen. Idempotency is the last line of defense, and it should be the first.

[celery]
broker_url = redis://redis:6379/0
result_backend = db+postgresql://airflow:airflow@postgres:5432/airflow
worker_autoscale = 12,3
worker_prefetch_multiplier = 1

[celery_broker_transport_options]
# Above the longest task runtime (6h ETL = 21600s)
visibility_timeout = 21600