Airflow Sensors: 4,320 Pokes and Still No File in Sight

A partner file lands between 1 AM and 7 AM. The pipeline needs it, so someone wrote a sensor to wait. Twelve hours later, that sensor had poked 4,320 times — once every 10 seconds — and the file still wasn't there. Worse, the wait had rented a full worker slot the entire time, and everything behind it queued.

Sensors are operators whose job is waiting. That's easy to write and easy to get wrong: the defaults that feel harmless — poke mode, no timeout, a tight poke interval — are exactly the settings that burn a slot for 12 hours and fail a DAG at dawn.

This article breaks down what sensors are, the worker-slot economics of poke vs reschedule, timeout and soft_fail semantics, the common sensor catalog, and the ExternalTaskSensor exactness trap. You'll also get the diagnosis flow for a sensor that waits too long.

If your sensor has been running for more than an hour, it's not waiting for the file. It's renting a worker slot to look at the clock.

1. What a Sensor Is: An Operator Whose Work Is Waiting

A sensor is an operator whose whole task is the wait: poke the condition, sleep the interval, poke again — until the condition is true or the timeout fires. Success means the condition became true; timeout means it didn't.

The condition can be a file on disk, an object in S3, a row in a database, a task in another DAG, or the clock itself. Each has a purpose-built sensor, and you can subclass BaseSensorOperator for anything else.

Design the wait like any other task: bounded, monitored, and cheap. The 4,320-poke night was none of those.

from airflow.decorators import dag
from airflow.providers.common.filesystem.sensors.file import FileSensor
from pendulum import datetime


@dag(schedule="@daily", start_date=datetime(2026, 8, 1), catchup=False, tags=["feeds"])
def feed_ingestion():
    wait_for_feed = FileSensor(
        task_id="wait_for_feed",
        filepath="/data/feeds/partner_orders.csv",
        poke_interval=60,
        timeout=3600,
        mode="reschedule",
    )


feed_ingestion()

2. poke vs reschedule: Worker Slot Economics

The mode decides where the wait happens. In poke mode — the default — the task stays in running state on a worker for the entire wait, sleeping between pokes. One sensor, one slot, for hours.

In reschedule mode, the task releases the slot between pokes and re-enters the queue. The poke still happens on schedule, but the wait no longer occupies a worker. Your parallelism budget stays yours.

The numbers from the incident: 10-second pokes for 12 hours is 4,320 pokes and one slot held for a full night. At 4 workers, that's a quarter of the fleet parked in front of a gate.

Defaults worth knowing: poke_interval defaults to 60 seconds, and the docs' rule of thumb is poke mode for sub-minute checks, reschedule for minute-level ones — the trade-off is latency, because every reschedule has to re-enter the queue.

3. timeout and soft_fail: Fail vs Skip vs Keep Waiting

timeout bounds the wait in seconds. Without it, a sensor pokes forever — the DAG never fails, the slot never frees, and the grid view shows a task that's been running for a week. Set timeout to the longest wait you can justify for that data.

soft_fail changes what a timeout means: the task is skipped instead of failed. That's the right call when the data is optional — a marketing feed you can live without — and the wrong call when the data is the reason the DAG exists.

Three outcomes, three designs: success when the file arrived, failed when it didn't and the data is mandatory, skipped when it didn't and the pipeline can proceed.

Two numbers frame the budget: the default timeout is seven days, and timeout counts from the first execution attempt, not per poke — so a rescheduled sensor's wall-clock lifetime can exceed it. When the bound does fire, the task raises AirflowSensorTimeout, which is exactly what soft_fail converts into a skip.

from airflow.providers.common.filesystem.sensors.file import FileSensor

wait_for_feed = FileSensor(
    task_id="wait_for_feed",
    filepath="/data/feeds/partner_orders.csv",
    poke_interval=60,
    timeout=21600,   # 6 hours — the maximum sane wait
    mode="reschedule",
)

# Optional data? Timeout becomes a skip, not a failure.
wait_for_marketing_feed = FileSensor(
    task_id="wait_for_marketing_feed",
    filepath="/data/feeds/marketing_emails.csv",
    poke_interval=120,
    timeout=3600,
    soft_fail=True,
)

4. The Common Sensor Catalog

You rarely write a sensor from scratch — the catalog covers the waits you'll actually meet. FileSensor watches a filesystem path, S3KeySensor watches for an object, SqlSensor polls until a query returns rows, TimeSensor and TimeDeltaSensor wait on the clock.

The one that confuses people is ExternalTaskSensor, which waits for a task instance in another DAG — it has its own section below because its exactness trap has cost teams a night or two.

When nothing fits, subclass BaseSensorOperator and implement poke(): return True when the wait is over. The table below is the catalog with the notes that matter.

The TaskFlow API has a shortcut too: the @task.sensor decorator turns any function that returns a PokeReturnValue into a sensor, which covers many quick waits without writing a class.

5. ExternalTaskSensor and Its Exactness Gotcha

ExternalTaskSensor waits for a task instance in another DAG to reach success. The gotcha: by default it matches the same data interval, not the same wall clock. If the upstream DAG runs on a different schedule or lands late, the sensor waits for a run that isn't the one you mean.

execution_delta shifts the match by a fixed amount — upstream schedules two hours later, offset by two hours. execution_date_fn handles anything irregular.

The failure shape is familiar: upstream succeeded hours ago, the sensor pokes forever, and the grid shows two healthy DAGs with one invisible mismatch between them.

from airflow.sensors.external_task import ExternalTaskSensor
from datetime import timedelta

wait_for_orders = ExternalTaskSensor(
    task_id="wait_for_orders_dag",
    external_dag_id="nightly_orders",
    external_task_id="load_orders",
    execution_delta=timedelta(hours=2),  # match the upstream's later schedule
    timeout=3600,
    mode="reschedule",
)

6. Write a Custom Sensor

Subclass BaseSensorOperator and implement poke(context): return True when the wait is over, False to poke again. Every BaseSensorOperator setting — poke_interval, timeout, mode, soft_fail — works unchanged.

Keep the poke cheap: it runs on every interval, so a poke that lists a directory is fine; a poke that scans a terabyte is a problem you've built yourself.

For the partner-file wait, the catalog's FileSensor already fits. Custom sensors earn their keep for domain conditions — a row reaching a state, an API returning a flag — that no provider sensor covers.

Two more knobs for long waits: exponential_backoff=True widens the gap between pokes on each failure, bounded by max_wait — the polite way to poll an external system with unpredictable availability.

from pathlib import Path

from airflow.sensors.base import BaseSensorOperator


class NewFileSensor(BaseSensorOperator):
    def __init__(self, filepath: str, **kwargs):
        super().__init__(**kwargs)
        self.filepath = filepath

    def poke(self, context):
        return Path(self.filepath).exists()


wait_for_partner_file = NewFileSensor(
    task_id="wait_for_partner_file",
    filepath="/data/feeds/partner_orders.csv",
    poke_interval=60,
    timeout=3600,
    mode="reschedule",
)

7. Reading the Grid When a Sensor Waits

A sensor in running state for 12 hours looks like a stuck task. The logs tell the truth: a Poking line every interval means the wait is alive and renting. Check three numbers — poke_interval, timeout, mode — before you suspect the scheduler.

The states help: up_for_reschedule in the grid means the sensor released its slot and is waiting cheaply. Running for hours with poke mode means the slot is held hostage.

And when the wait genuinely needs to be long — hours, not minutes — skip the slot question entirely: deferrable sensors hand the wait to the triggerer process, where it costs almost nothing (that's the deferrable-operators article, and it's the right upgrade path for every sensor in this one).