Airflow Monitoring: Dead Scheduler Nobody Saw for 9 Days

Nobody checks the scheduler on a quiet Tuesday. DAGs go green, the Grid view looks calm, and the team ships features. That's exactly when a dead scheduler hides best.

One team learned this the expensive way. Their scheduler died during a routine node rotation, every DAG silently stopped, and nine days passed before finance asked where the revenue report went. No page. No Slack ping. Nothing.

This guide shows you the three heartbeat metrics that catch that failure in minutes, how to ship logs to S3 or GCS so evidence survives pod evictions, and how SLA misses become first-class alerts. Nine days. Zero runs.

The Three Metrics That Prove Airflow Is Alive

Three numbers tell you Airflow is alive. Scheduler heartbeat age shows the scheduler wrote to the job table recently. Queued-task age shows tasks aren't starving in pools. SLA miss count shows business promises still hold.

Heartbeat age is the king metric. The scheduler emits it every few seconds, and anything over 5 minutes stale means scheduling stopped, whether the process died or wedged on a DB lock. You'll graph it as time minus last heartbeat timestamp.

Queue age catches what heartbeats miss. A live scheduler with a wedged pool still schedules nothing useful, and p95 queued-task age per pool exposes exactly that. Track depth too, but page on age.

Pick your metrics pipe on purpose. StatsD is the long-standing path: set statsd_on = True with host, port 8125, and a prefix, then bridge UDP to Prometheus with statsd_exporter and mapping rules that turn DAG- and task-embedded names into labels. OpenTelemetry is the forward-looking path in Airflow 3: flip otel_on in [metrics], point at your OTel Collector, and let it route to Prometheus. If you're starting fresh, OTel's ecosystem wins long-term. One gotcha either way: StatsD drops tag mappings by default, so enable a tagged wire format (statsd_influxdb_enabled or statsd_datadog_enabled) or your per-DAG labels never materialize in Prometheus.

Don't skip the /health probe. The API server's /health endpoint reports the metadata DB plus fresh heartbeats for scheduler, triggerer, and DAG processor, and it's what your liveness checks should hit. Tie it to metrics: fire an Alertmanager rule when scheduler-heartbeat lag crosses 60s so you don't hand-refresh a health page at 3 AM. Five signals stay core: heartbeat, queue backlog, failure rate, parse time, and pool utilization — a pool pinned at 100% with waiters piling up means its slots need a recount, not a bigger dashboard.

groups:
- name: airflow-scheduler
  interval: 30s
  rules:
  - alert: AirflowSchedulerDead
    expr: time() - airflow_scheduler_heartbeat_timestamp > 300
    for: 2m
    labels:
      severity: page
      team: data-platform
    annotations:
      summary: Scheduler heartbeat stale over 5 minutes
      runbook: https://wiki.acme.io/runbooks/airflow-scheduler-dead
  - alert: AirflowQueueStarving
    expr: histogram_quantile(0.95, airflow_queued_task_age_seconds) > 600
    for: 5m
    labels:
      severity: page
      team: data-platform
    annotations:
      summary: P95 queued-task age above 10 minutes
  - alert: AirflowSLAMiss
    expr: increase(airflow_sla_missed_total[15m]) > 0
    labels:
      severity: ticket
      team: dag-owner
    annotations:
      summary: SLA miss on a tier-1 DAG in the last 15 minutes

Shipping Task Logs to S3 and GCS

Local task logs die with the worker. On Kubernetes a pod eviction wipes the only copy of the traceback you need, and even on VMs a disk-full event takes logs with it. Remote logging fixes this by writing every task log to object storage as it streams.

Setup takes four settings. Point REMOTE_BASE_LOG_FOLDER at your bucket, give workers a conn id with write-only IAM permissions, and turn on DELETE_LOCAL_LOGS with a 7-day interval so disks don't fill. The UI reads remote logs transparently, so developers notice nothing except logs that never vanish.

Verify before you trust it. Run one task, list the bucket prefix, and confirm the log object exists. Then kill a worker mid-task on staging and confirm the log survives. You'll thank yourself during the next real eviction.

# airflow.cfg logging section for production log shipping
AIRFLOW__LOGGING__REMOTE_LOGGING=True
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=s3://acme-airflow-logs/prod
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=aws_logs_writer
AIRFLOW__LOGGING__DELETE_LOCAL_LOGS=True
AIRFLOW__LOGGING__DELETE_LOCAL_LOGS_INTERVAL=7

# verify from any host
firstname=$(airflow config get-value logging remote_base_log_folder)
echo "remote logs go to: $firstname"
aws s3 ls s3://acme-airflow-logs/prod/ --human-readable | head -20

SLA Misses as First-Class Alerts

SLA misses are the only alert the business understands. Tasks can retry their way to success 4 hours late while every technical dashboard stays green. The sla parameter draws the line: this DAG must finish within 2 hours of its run, or someone hears about it.

Wire sla_miss_callback to the team that owns the outcome, not a general channel. Finance DAGs page finance-adjacent engineers in #data-incidents; experimental DAGs file tickets. Routing matters because an SLA alert the wrong team ignores is the same as no alert.

Start strict on tier-1 DAGs only. Five SLA alerts on critical pipelines beat fifty noisy ones everyone mutes. You'll tune thresholds with a month of data, and each adjustment gets a note in the runbook.

Heads-up for Airflow 3: classic SLA monitoring is gone. The old sla_miss table and sla_miss_callback behavior was removed because its logical_date math confused everyone. The replacement is the Deadline concept — you state the intent explicitly (reference time plus grace period, then run this callback) instead of inheriting N-minutes-from-logical-date. If you're migrating a 2.x DAG, don't port sla= lines one-for-one; redesign each alert on Deadline and confirm the exact API shape in the airflow.apache.org docs for your 3.x minor, since this area is still settling version to version. Keep the same rule: a lateness record nobody pages on is decoration.

import datetime
import pendulum
from airflow.sdk import dag
from airflow.providers.slack.notifications.slack import send_slack_notification

BUSINESS_SLA = datetime.timedelta(hours=2)

sla_alert = send_slack_notification(
    slack_conn_id="slack_data_alerts",
    text="SLA miss on {{ dag.dag_id }}: run {{ run_id }} is late.",
    channel="#data-incidents",
)

@dag(
    dag_id="revenue_daily",
    schedule="0 6 * * *",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    sla=BUSINESS_SLA,
    sla_miss_callback=sla_alert,
    tags=["tier-1", "finance"],
)
def revenue_daily():
    pass

revenue_daily()

PagerDuty, Slack, and Email That Actually Fire

One channel per severity keeps alerts actionable. Pages go to PagerDuty for heartbeat and queue-age failures. Slack gets SLA misses and failure-rate warnings where the owning team already lives. Email digests cover daily summaries nobody needs at 3 AM.

Keep the signal path short. Airflow notifiers post straight to Slack webhooks, and a small PagerDuty integration turns repeated SLA misses into incidents. You'll avoid the classic trap of alerts landing in a channel with 200 members and zero owners.

Test the path monthly. A rotated Slack webhook silently breaks notifications, which recreates the 9-day silence with better intentions. Send a test alert from staging after every credential rotation.

Anatomy of the 9-Day Silent Failure

The 9-day outage had a boring anatomy. Node rotation killed the scheduler, the job-table heartbeat froze, and run creation stopped. Depth metrics stayed flat because flat looks healthy when nothing arrives. The webserver masked everything by serving a perfect UI over stale data.

Three alerts would have caught it in minutes. Heartbeat age pages at 5 minutes stale. Queue age pages when the first tasks starve. An SLA miss on the revenue DAG fires when the business promise breaks. Total cost: two Prometheus rules and one callback.

The lesson isn't more dashboards. It's alerting on absence: no heartbeat, no runs, no logs shipped. You'll catch the next silent failure by watching for what should exist but doesn't.

Runbooks That Survive On-Call Rotations

A runbook turns a 3 AM page into a checklist. One page per alert: symptom, first three commands, escalation path, and the rollback step. The scheduler-dead runbook starts with airflow jobs check and ends with the restart command plus verification.

Keep runbooks next to the alert. Link the wiki page in the Prometheus annotation so the paged engineer opens it in one click. You'll cut mean-time-to-recovery from an hour of guessing to ten minutes of following steps.

Review runbooks after every incident. If a step didn't help, delete it. If a command was missing, add it. A runbook that survives three incidents unchanged is either perfect or unread, and it's never perfect.