Airflow Celery: Dead Workers Nobody Noticed Until Dawn

The queue looked fine until it wasn't. Tasks sat queued for hours while Flower showed workers that had died silently overnight.

Nobody restarted them because nothing alerted. Celery doesn't resurrect workers on its own; dead is dead until a human or a healthcheck intervenes.

Celery decouples execution from scheduling, which scales beautifully right up until the fleet needs babysitting. You'll learn that babysitting here.

Why Celery: Decoupled Task Execution

Celery splits scheduling from execution. The scheduler writes task messages to a broker instead of forking processes locally. Workers on any host consume those messages and run tasks, reporting results back. That decoupling is what lets you scale from one box to twenty.

The price is a distributed system: broker, workers, result backend, and Flower all need operating. LocalExecutor's simplicity disappears the day you switch, replaced by real capacity that needs real babysitting.

Choose Celery when steady throughput exceeds one box. Keep LocalExecutor while one box still drains peaks comfortably.

Broker Setup: Redis vs RabbitMQ

Install the provider first: pip install 'apache-airflow[celery]' (or apache-airflow-providers-celery>=3.3.0) plus librabbitmq or redis client libs. Redis is simpler to operate: one process, fast, familiar persistence tuning. RabbitMQ offers durable quorum queues and richer routing at the cost of heavier clustering ops, with Redis Sentinel covering HA without a full cluster. Both carry Celery fine; durability requirements pick the winner.

Set broker_url and result_backend explicitly, never defaults, and prefer a database-backed result backend so broker cleanup and result storage stay separate. Redis needs AOF persistence so restarts don't amnesia the queue. RabbitMQ needs quorum queues where zero-loss matters. Three rules hold across fleets: airflow plus the CLI on every worker's PATH, homogeneous Airflow config on every box, and operator dependencies (Hive CLI, DB libs, PYTHONPATH) present where tasks run.

Workers need their DAGS_FOLDER synced by your own means: Git plus Chef, Puppet, or Ansible, or a shared mount every box sees. Load-test the broker before prod. A broker that handles dev's 10 tasks per minute can collapse at prod's 500, and scheduler publish errors look exactly like scheduler bugs. Queue names cap at 256 characters, and each broker trims further, so keep names short.

[core]
executor = CeleryExecutor

[celery]
broker_url = redis://redis:6379/0
result_backend = db+postgresql://airflow:airflow@postgres/airflow
worker_concurrency = 8
worker_autoscale = 12,4
worker_prefetch_multiplier = 1
broker_transport_options = {"visibility_timeout": 21600}

# start an etl-lane worker
# airflow celery worker -Q etl,default --concurrency 8 --autoscale 12,4

Workers, Queues, and Autoscale

Workers consume from queues with bounded concurrency. Autoscale flexes between min and max with queue depth: --autoscale 12,4 means up to 12 slots under load, down to 4 idle. Static concurrency wastes money idle and starves busy. Start lane-specific workers with airflow celery worker -q spark,quark (comma-delimited, no spaces); tasks route via queue='etl' and the default_queue in airflow.cfg catches the rest.

Queues separate lanes: etl for warehouse loads, ml for training, default for checks. Start workers subscribed per lane so capacity follows labels. Prefetch multiplier 1 stops workers hoarding tasks they can't start. Monitor worker_concurrency against box RAM: each slot is a forked task process.

Drain before restarts. A worker killed with 8 running tasks requeues all 8, and non-idempotent tasks double-write on redelivery. Stop gracefully with airflow celery stop (SIGTERM per Celery docs), never kill -9 on a loaded box unless you've rehearsed redelivery.

Flower for Fleet Visibility

Flower shows workers, queues, task rates, and unacked counts via airflow celery flower. It's a debug view, not monitoring: nobody stares at it at 3 AM. Put queue-depth and worker-heartbeat alerts in your paging system and use Flower for diagnosis after the page. Install the flower package from the celery bundle first, and emit JSON worker logs with [celery] json_logs=True when your log stack wants structure.

Run Flower behind auth with restricted network access. It exposes task args that may include table names and partition keys you'd rather not publish.

Pair Flower with queued-task age dashboards. Age climbing while workers show idle is the signature of prefetch, pool, or timeout misconfiguration. Treat Redis as transient broker state: never flush its keys while schedulers or workers run. To discard stale broker data, stop everything, confirm no queued or running tasks matter, back up persistent Redis, and clear only the keyspace in [celery] broker_url. Metadata history lives in the DB and needs airflow db clean instead.

# worker liveness probe (container or systemd)
# airflow celery worker -Q etl --concurrency 8 --autoscale 12,4

# Flower behind auth, not open to the internet
# airflow celery flower --basic-auth "ops:${FLOWER_PASSWORD}" --port 5555

# queue-depth alert (cron every 2 min)
# redis-cli LLEN celery || rabbitmq-diagnostics check_port_connectivity
# page when pending > 500 for 10 minutes

Worker Death: Detect, Restart, Drain

Worker death has three acts: OOM or crash, no restart policy, silent queue growth. Detect via heartbeat alerts and restart via platform policy (systemd, container restart, or ASG replacement). Drain survivors before re-adding capacity so redelivered tasks don't stampede.

Visibility timeout governs redelivery: the broker requeues unacknowledged tasks after the timeout. Set it above your longest task plus buffer or long jobs execute twice. That duplicate is a correctness bug, not extra throughput.

Rehearse death in staging: kill -9 a worker mid-task, watch redelivery, confirm idempotent writes absorb it. Prod is the wrong place to meet your broker's redelivery behavior.

The Visibility-Timeout Double-Execution Trap

The trap is subtle: the first copy still runs while the broker, assuming death, hands the task to another worker. Both copies write the same partition minutes apart. Logs show two successes; the table shows doubled rows.

Fix the timeout first, idempotency second, alerting third. Timeout stops the duplicates, partition-keyed writes absorb the survivors, and redelivered-count alerts catch regressions.

Document your longest tasks. The timeout you set today rots the day someone ships a 5-hour backfill task.