Airflow Sensors Exposed: 4,320 Pokes and No File Found

Your FileSensor poked 4,320 times overnight and the file never arrived. It also held a worker slot the whole time, so real tasks queued behind a wait.

Sensors are not free. You'll learn when to poke, when to reschedule, and when to defer so waits stop costing you workers.

We cover the sensor catalog, timeout math, and the custom sensor pattern that keeps prod calm. You'll set timeouts on every sensor after this.

Waiting should be cheap. Make it so.

What a Sensor Is: An Operator That Waits

A sensor succeeds when its condition appears and fails when its timeout expires. Until then it pokes on an interval you control.

Use sensors for files, partitions, upstream DAGs, and API readiness. Do not use them as polling ETL loops; they wait, they do not extract.

from airflow.decorators import dag, task
from airflow.sensors.filesystem import FileSensor
from datetime import datetime

@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False, tags=["ledger"])
def ledger_daily():
    wait_file = FileSensor(
        task_id="wait_vendor_file",
        filepath="/data/landing/vendor_ledger.csv",
        poke_interval=300,
        timeout=21600,
        mode="reschedule",
        soft_fail=False,
    )

    @task
    def load_ledger():
        return {"rows": 88000}

    wait_file >> load_ledger()

ledger_daily()

Poke vs Reschedule: Worker Slot Economics

Poke holds a worker slot for the entire wait and sleeps between checks. Reschedule frees the slot between checks and re-queues the poke.

Short waits under a minute can poke. Anything longer should reschedule. A 12-hour poke at 10-second intervals burned 4,320 slot-minutes for nothing.

Latency decides the mode: checks every few seconds belong in poke, checks every minute or slower belong in reschedule. Reschedule with a poke_interval under about 5 minutes can swamp the scheduler with rapid re-queues, so lengthen the interval instead of shrinking it. For waits past an hour, add deferrable True where the sensor supports it and the Triggerer watches while workers stay free. Set exponential_backoff True with max_wait to back off polling against flaky vendors instead of hammering them at a fixed cadence.

Timeout and Soft Fail: Fail vs Skip vs Keep Waiting

timeout caps the total wait from first poke to deadline. When it expires the sensor fails unless soft_fail is true, which marks it skipped.

Use fail for money-path inputs where missing data must page. Use soft_fail for optional partitions where downstream should skip gracefully.

The stock timeout is seven days, which is a trap; always set an explicit deadline per dataset. timeout counts from the first poke attempt, not per retry, and retries restart parts of that accounting, so a sensor with generous retries can outlive its timeout across tries. Don't confuse timeout with execution_timeout: in reschedule mode each poke attempt is short, so execution_timeout barely bites while timeout governs the whole wait. Two more failure flavors exist: silent_fail logs poke exceptions and keeps waiting, while soft_fail converts deadline misses to skips. A proven pattern pairs soft_fail with a BranchPythonOperator on ALL_DONE: the sensor skips on timeout, the branch reads the sensor state and routes to a with-data or without-data path.

from airflow.decorators import dag, task
from airflow.sensors.filesystem import FileSensor
from datetime import datetime

@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False, tags=["ledger"])
def ledger_optional():
    wait_optional = FileSensor(
        task_id="wait_marketing_export",
        filepath="/data/landing/marketing.csv",
        poke_interval=600,
        timeout=10800,
        mode="reschedule",
        soft_fail=True,
    )

    @task(trigger_rule="none_failed")
    def summarize():
        return {"status": "done"}

    wait_optional >> summarize()

ledger_optional()
# soft_fail=True turns timeout into skipped; pair with BranchPythonOperator(TRIGGER ALL_DONE)
# to route to a no-data path instead of paging at 3 AM

The Common Sensor Catalog

FileSensor watches landing paths. S3KeySensor watches object keys. ExternalTaskSensor waits on another DAG run. TimeSensor waits until a clock time.

SqlSensor polls a row count or watermark. HttpSensor checks API readiness. Pick the sensor that matches the event instead of shelling out in Bash.

Rough starting points that survive prod: FileSensor at 60s poke with a 2 to 4 hour timeout, ExternalTaskSensor at 300s poke with 6 to 12 hours, HttpSensor at 30 to 60s with 30 minutes to 2 hours, S3KeySensor at 60s with 2 to 4 hours, SqlSensor at 30s poke with 30 minutes to an hour. Prefer PythonSensor over BashSensor for JSON or auth logic, and TimeDeltaSensor over sleep calls so waits stay visible in the UI. HttpSensor needs its own request timeout separate from the sensor timeout or one hung socket eats the whole deadline.

ExternalTaskSensor and Its Exactness Gotcha

ExternalTaskSensor needs the exact logical date of the upstream run. A one-hour delta mismatch means it waits forever on a run that already succeeded.

Pin external_dag_id, allowed_states, and execution_delta carefully. When producers and consumers drift, switch to asset-triggered scheduling instead of date math.

from airflow.decorators import dag, task
from airflow.sensors.external_task import ExternalTaskSensor
from datetime import datetime, timedelta

@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False, tags=["ledger"])
def ledger_consumer():
    wait_upstream = ExternalTaskSensor(
        task_id="wait_upstream",
        external_dag_id="vendor_daily",
        external_task_id="publish_ledger",
        allowed_states=["success"],
        execution_delta=timedelta(hours=1),
        poke_interval=300,
        timeout=14400,
        mode="reschedule",
    )

    @task
    def reconcile():
        return {"status": "reconciled"}

    wait_upstream >> reconcile()

ledger_consumer()

Write a Custom Sensor

Subclass BaseSensorOperator and implement poke() returning True or False. Return False for not-ready, True for ready, and raise only on auth failures.

Catch transient network errors inside poke and return False so retries handle blips. Keep poke fast under 30 seconds and idempotent across calls.

Modern Airflow also offers @task.sensor, which turns a Python function into a sensor returning PokeReturnValue(is_done=..., xcom_value=...). The xcom_value lands in XCom on success, so the waiter can hand the found partition path straight downstream with zero extra tasks. Keep poke under 30 seconds, catch transient errors and return False, and reserve raises for auth failures.