Airflow Deferrable Operators: Cut Sensor Costs to Zero

Worker slots are expensive. Every sensor sleeping in poke mode rents one by the minute while doing absolutely nothing. Multiply by dozens of waiting sensors and your cluster works hard at standing still.

One file sensor held a full slot for 40 minutes waiting on data that arrived late. Real tasks queued behind it, SLAs slipped, and the fix cost one keyword argument. You'll learn which one.

Deferrable operators hand the waiting to the triggerer and free the slot. Pennies instead of dollars. Let's do the math.

The Poke Problem: Worker Slot Economics

A poke sensor rents a worker slot and naps on it. Between checks it sleeps, but the slot stays reserved, billed, and unavailable to real work. A one-second check repeated for 40 minutes costs what 40 minutes of compute costs.

You'll feel it as phantom load. Worker occupancy reads 100%, CPU reads 15%, and ready tasks queue behind sensors that are technically running. Adding workers helps briefly, then the new slots fill with more napping sensors.

Price waits in slot-minutes. One 40-minute poke wait burns 40 slot-minutes; fifty concurrent ones burn 2,000. That single number usually ends the debate about whether sensor modes matter.

Deferrable Model: Defer Plus Resume

Deferrable flips the model. The task suspends itself, releases its worker slot, and registers an async trigger. When the condition fires, the task resumes through the queue like any other work. Waiting moves from workers to the triggerer.

You'll convert with one argument. deferrable=True on supported sensors plus a running triggerer is the whole migration for most waits. The Grid view shows the honest deferred state instead of a fake running one.

Keep the safety rails. Timeouts still bound the wait, poke_interval still spaces trigger checks, and mode='reschedule' remains a sane fallback. Deferral changes where waiting happens, not whether deadlines apply.

Flip fleets with one setting when you're ready. [operators] default_deferrable (env AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE) makes every sensor/operator that supports both modes defer by default, so deferrable=True stops being a per-task negotiation. Override per task where a tight local poke still makes sense. And know the two worker-skipping upgrades since 2.10: start_from_trigger defers straight to the triggerer without ever renting a worker, and end_from_trigger finishes there too — no resume slot at all for pure waits.

import datetime
import pendulum
from airflow.sdk import dag, task
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor

@dag(
    dag_id="landing_guard",
    schedule="*/15 * * * *",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    tags=["sensors", "landing"],
)
def landing_guard():
    wait_for_drop = S3KeySensor(
        task_id="wait_for_drop",
        bucket_name="acme-landing",
        bucket_key="sales/{{ ds }}/_SUCCESS",
        deferrable=True,  # free the slot; triggerer watches instead
        poke_interval=60,
        timeout=datetime.timedelta(hours=2),
        mode="reschedule",  # fallback path if triggerer is down
    )

    @task
    def promote() -> str:
        return "drop present, promoting"

    wait_for_drop >> promote()

landing_guard()

The Triggerer Process and Its Scaling

The triggerer is one async event loop watching thousands of conditions. It costs a fraction of a worker and replaces whole fleets of napping sensors. You'll run it as a first-class component, not an afterthought.

Scale it on backlog, not vibes. The queued-trigger count and resume latency tell you when one triggerer saturates; a second triggerer or bigger host follows the same playbook as scheduler scaling. Idle triggers are cheap, simultaneous resume bursts are the load to plan for.

Monitor it like workers. Heartbeat, backlog, resume latency, and failure rate each get a dashboard panel and one alert. A dead triggerer parks every deferred task silently, which recreates the exact outage deferrable was bought to prevent.

Size it with the real knob: triggerer capacity defaults to 1000 concurrent triggers per process (config triggerer capacity, env AIRFLOW__TRIGGERER__DEFAULT_CAPACITY). Run two-plus replicas for HA like schedulers — triggers survive in the metadata DB across restarts, but nothing resumes while zero triggerers run. Watch the triggerer heartbeat like any component, and use queues_enabled with --queues on the triggerer hosts when teams need isolated trigger lanes.

Converting Sensors With Deferrable True

Conversion is mechanical. Find sensors with long timeouts in the DAG folder, add deferrable=True where the provider supports it, and confirm the deferred state on the next run. You'll batch the change per team to keep reviews small.

Verify provider support first. Most file, S3, and external-task sensors support deferrable in current providers; exotic ones may not yet. Unsupported sensors move to mode='reschedule' as the interim fix, not eternal poke.

Gate it in review. Any new sensor with an expected wait over 5 minutes must justify poke mode in the pull request. Defaults drift back to poke without the rule written down.

Cost Math: 40-Minute Poke Versus Zero

The incident math is stark. One 40-minute poke wait costs 40 slot-minutes; deferred it costs under one. Fifty concurrent waits cost 2,000 slot-minutes poking versus a single triggerer's spare capacity deferred. That's a whole worker fleet versus a sidecar.

You'll translate it to money per quarter. Slot-minutes times worker cost per minute times waiting sensors per day compounds into real budget. Teams that price it once never go back to poke for long waits.

Count the triggerer honestly. It needs its host, its monitoring, and burst headroom. Even fully loaded it costs an order of magnitude less than the slots it frees, but free is a lie and budgets deserve the truth.

# run the triggerer alongside the scheduler (one per cluster to start)
airflow triggerer &
ps aux | grep -i triggerer | grep -v grep

# how many tasks currently wait on the triggerer?
# (metadata DB check during business hours)
PG="postgresql://airflow:***@db:5432/airflow"
psql "$PG" -c "SELECT state, count(*) FROM task_instance WHERE state IN ('deferred','scheduled') GROUP BY 1;"

# cost math for the review doc
python3 -c "print('poke cost per 40-min wait: 1 slot x 40 min =', 40, 'slot-minutes')"
python3 -c "print('deferred cost per 40-min wait: ~0 slots + triggerer share =', '<1', 'slot-minute')"

Timeouts and Resume Semantics

Timeouts bound every wait, deferred or not. A file that never lands must fail the task, page the owner, and free the DAG run. Without a timeout the run parks in deferred forever and the SLA misses with no failure to investigate.

Resume semantics reward idempotent downstream tasks. A resumed task re-enters the queue and may run on a different worker minutes later; anything it touches must tolerate that gap. You'll design post-sensor tasks like post-retry tasks.

Test the deadline path. Set a one-minute timeout on staging, run the sensor against a missing key, and confirm it fails loudly with the right alert. Untested timeouts are wishes, and wishes don't page.

Write triggers with triggerer rules in mind. Run() must be an async generator yielding TriggerEvent — and every client inside it must be async (aiohttp, aiobotocore), never blocking SDK calls, or one trigger stalls the whole event loop. Keep constructor args JSON-serializable or defer blows up at serialize time. Put external cancellation (kill that BigQuery job, terminate that Databricks run) in on_kill, which fires on explicit user kills but never on triggerer restarts or rolling deploys — cleanup-style logic there would cancel in-flight work mid-deploy.