Airflow Monitoring: The Dead Scheduler Nobody Saw!

The scheduler is the heart of Airflow. It parses DAGs, creates runs, and hands tasks to the executor — and when it dies, the whole platform goes quiet without a single error. We found that out the hard way. Our scheduler process crashed, and for nine days the metadata DB kept answering health checks while the DAGs sat dark.

Monitoring Airflow isn't about pretty dashboards. It's about five metrics that tell you the platform is alive, processing, and on time: scheduler heartbeat, parse lag, queue depth, task age, and SLA misses. Two of those — the heartbeat and the SLA miss — are the ones everyone skips.

Logs are the other half. Task logs written to the local disk vanish when a worker dies or a pod is evicted; shipping them to S3 or GCS means you can still debug the run that failed at 3 AM.

A dead scheduler is a binary event. Your monitoring should treat it like one.

1. The Five Signals That Matter

Monitoring Airflow is not a dashboard contest. Five signals tell you the platform is alive, on time, and producing the logs you'll need when it isn't.

The heartbeat measures scheduler liveness. Parse lag tells you how far behind the scheduler is. Queue depth shows work waiting for an executor. Task age exposes tasks that should have finished. SLA misses catch data that's late even when DAGs run green.

Everything else — CPU, memory, disk — is infrastructure noise until one of these five breaks.

Five signals, five alerts. That's the whole practice.

Airflow emits these over StatsD — statsd_on = True, statsd_host, statsd_port = 8125, statsd_prefix — or, in Airflow 3, OpenTelemetry. A statsd_exporter bridges the UDP stream to Prometheus scrape endpoints, and its mapping config is what turns Airflow's dag- and task-embedded metric names into queryable labels.

Schedule delay — the gap between a DAG's scheduled start and its actual start — is the leading indicator in the set: it starts rising before the executor saturates, while everything else still looks healthy.

# The five signals as Prometheus queries
scheduler_heartbeat: max(time() - airflow_scheduler_heartbeat_last_finished{})
parse_lag: max(time() - airflow_dag_processing_last_run_seconds_ago{})
queue_depth: sum(airflow_executor_queued_tasks{})
task_age: max(time() - airflow_task_instance_created{})
sla_misses: sum(airflow_sla_missed_dags{})

2. Scheduler Heartbeat: The Life Signal

The scheduler reports its heartbeat on every loop it completes. The metric stores the timestamp of the last finished cycle. The moment the process dies — or stalls — that timestamp stops moving.

Our heartbeat was never scraped. The webserver kept answering, the DB kept responding, and the scheduler metric aged like a corpse nobody checked. Heartbeat staleness is the single highest-value alert in Airflow.

Set the alert at three missed beats. With a default heartbeat every 5 seconds, that's roughly 15 seconds to a page.

Three beats. Fifteen seconds. One page.

The triggerer has its own heartbeat metric, triggerer_heartbeat — while it's down, deferred tasks accumulate — so give it the same staleness alert. And Airflow's scheduler_heartbeat counter is a rate metric: alert when its rate of change drops to zero, the dead-process signature.

3. Shipping Task Logs to Object Storage

Task logs land on the worker's local disk by default. Local disk is ephemeral: workers get recycled, pods get evicted, containers get recreated — and the logs go with them.

Remote logging writes every task log to S3, GCS, or Azure Blob at the end of each task. The overhead is small — roughly 40ms per task on a typical run — and the payoff is huge: you can debug any failed run from any month, on any machine.

Remote logging is the difference between a post-mortem and a shrug.

[logging]
remote_logging = True
remote_base_log_folder = s3://airflow-logs-prod/logs
remote_task_handler = airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler

4. SLA Misses as First-Class Alerts

SLA means the latest acceptable finish time for a DAG run. Airflow records a miss in the sla_miss table when a run crosses it — but recording is all it does unless you wire the callback.

Set sla in default_args, point sla_miss_callback at a function that pages the on-call, and a late DAG becomes a human problem in minutes instead of a user problem in days.

Our dead scheduler produced nine days of SLA misses. The table recorded every one. Nobody was looking at the table.

from datetime import timedelta
from airflow.models.dag import DAG


def on_sla_miss(context):
    dag_id = context["dag"].dag_id
    notify_pagerduty(
        incident_title=f"SLA miss: {dag_id}",
        description=(
            f"{dag_id} missed its SLA for run "
            f"{context['execution_date']}"
        ),
    )


default_args = {
    "owner": "data-eng",
    "sla": timedelta(hours=3),
    "sla_miss_callback": on_sla_miss,
}

5. Notification Integrations: PagerDuty, Slack, Email

Alerts are only as good as their routing. Email buries you, Slack channels scroll, PagerDuty wakes a human. A sane stack uses all three with different severities.

Pages for heartbeat staleness, queue-depth overflow, and SLA misses. Slack for warnings and DAG-level failures. Email for the digest that nobody reads but compliance likes.

The rule of thumb: if an alert doesn't page, it's a log entry, not an alert.

6. The 9-Day Silent Failure: Alert Anatomy

Tuesday, 02:14 — the scheduler container ran out of memory and died. Nothing else changed. The webserver kept serving, the UI kept rendering, and the metadata DB kept answering health probes.

Day 1: the first queued task ages past an hour. No alert — task age wasn't monitored.

Day 3: the first SLA miss lands in sla_miss. No callback — it was never wired.

Day 9: a user asks why yesterday's dashboard is stale. That's when the outage ends — not with a page, but with a ticket.

The alert that should have existed is a single rule: heartbeat age over 60 seconds. It would have fired at 02:15, one minute after the process died.

7. Runbooks That Survive

An alert without a runbook is a panic with extra steps. When the page fires at 3 AM, the person on call needs a checklist: confirm the signal, check the obvious causes, restart safely, verify recovery.

Write the runbook when you write the alert. Two paragraphs, real commands, exact expectations. Our dead-scheduler runbook now says: check heartbeat age, pull scheduler logs, restart the container, verify the next heartbeat lands within 60 seconds.

A runbook that survives is one that's been executed in a drill. Run the drill on a Tuesday afternoon, not during the incident.