Airflow HITL: The Schema Change That Skipped Review

Every team that runs destructive DDL in production eventually has the same nightmare. A migration task that was 'scheduled for the night window' executes unattended, a breaking schema change ships, and the warehouse degrades for the next week — while everyone was asleep.

Airflow's answer is human-in-the-loop: a task that pauses the DAG and asks a real person to approve or deny before the dangerous step runs. In Airflow 3 it's a first-class citizen — the HITLOperator — with its own UI, notifications, and timeout semantics.

The pattern is older than Airflow. Deploy tools, change boards, and money systems have gated approvals for decades; orchestration platforms are finally catching up.

The trap is building gates that can be skipped, expired, or ignored.

HITL isn't bureaucracy. It's the difference between a review that happens and a review that's supposed to happen.

1. Why Human Gates Exist in Pipelines

Some operations are cheap to reverse and some aren't. Dropping a column, deploying to production, moving money — the blast radius is wide, the rollback is painful, and the context needed to make the call is precisely what a pipeline doesn't have. Automation guarantees the operation runs correctly. It can't guarantee the operation should run at all.

That's the argument for human gates: a point in the DAG where progress requires a judgment call that a schedule can't make. Code review works the same way — a second set of eyes before the irreversible step, with a record of who signed off.

The failure mode to fear isn't slowness. It's theater — gates that exist on paper but are skippable in practice, approved by rote, or bypassed 'just this once' because the branch was simpler.

2. HITLOperator: The Approve/Deny Task

Airflow 3 ships HITLOperator, an operator that defers the pipeline until a human responds. It appears in the UI with its own approve and deny actions, so an operator on call can act without leaving the browser. On approve, the DAG proceeds to the next task; on deny, the gate fails and downstream tasks don't run.

The operator defers like a deferrable operator — no worker slot is burned while waiting — and the approval event resumes the task. That means a gate can wait for hours without consuming executor capacity, which is exactly what an overnight approval window needs.

Treat the gate as the controlling task: wire everything dangerous downstream of it, and wire nothing that could bypass it. The approval is a binary decision with a comment; make sure the comment is captured, because it's the 'why' your post-mortem will ask for.

HITL is a family, not one operator (Airflow 3.1+, standard provider): HITLOperator is the base class for arbitrary options, ApprovalOperator restricts to Approve/Reject, HITLBranchOperator routes to a downstream task of the human's choice, and HITLEntryOperator collects free-form input through a params form — the standard tool for LLM workflows that need human guidance. The decision travels as XCom: chosen options land under the chosen_options key and form input under params_input, which downstream tasks pull to branch on or record. One semantic to know: ApprovalOperator treats Reject as success-with-skip — the gate task succeeds and everything downstream is skipped; pass fail_on_reject=True when a rejection must fail the run.

Pending gates surface on the UI's Required Actions page, and the same requests are reachable through the REST API — GET /api/v2/hitlDetails/ lists them, PATCH /api/v2/hitlDetails/{dag_id}/{dag_run_id}/{task_id} submits the answer — so approvals work without a browser, from a script or a Slack app. In Airflow 3.3+, a waiting HITL task uses a dedicated scheduler-managed awaiting_input state that holds no worker slot and no triggerer (the triggerer can scale to zero while gates wait); on 3.1/3.2 it defers onto the triggerer.

from datetime import datetime, timedelta

from airflow import DAG
from airflow.operators.hitl import HITLOperator
from airflow.providers.postgres.operators.postgres import PostgresOperator

with DAG(
    dag_id="migration_orders_v2",
    schedule="0 3 * * *",
    start_date=datetime(2026, 1, 1),
    catchup=False,
) as dag:
    approval = HITLOperator(
        task_id="approve_schema_change",
        notification_webhook_url=SLACK_WEBHOOK_URL,
        message="Approve dropping column legacy_promo from prod.orders?",
        timeout=60 * 60 * 6,  # 6 hours, aligned with on-call SLA
    )

    run_migration = PostgresOperator(
        task_id="run_migration",
        postgres_conn_id="postgres_prod",
        sql="ALTER TABLE prod.orders DROP COLUMN legacy_promo;",
    )

    approval >> run_migration

3. Wiring Notifications: UI, Slack, Email

A gate nobody sees is a gate nobody approves. The HITLOperator supports a notification webhook, and the integration of choice in 2026 is Slack — the approval message lands in the channel where the on-call engineer actually lives. Email is a fallback, not a strategy; inboxes are where approvals go to die.

Route to a named approver or a rotation, never a shared inbox. 'Data platform' channels are where everyone assumes someone else will click. The gate message must name the person on call, include the context (what changes, what breaks if wrong), and link back to the DAG run.

Escalation is the second half of notification design. If no one responds within a fraction of the timeout, the gate should page the next tier. A single message that goes unanswered for six hours is a stalled pipeline; a message that escalates at 30 minutes becomes a decision.

The operator API wires this through notifiers: a list of callables invoked when a request starts waiting, succeeds, or fails (the docs' LocalLogNotifier shows the interface; a notifier that POSTs to a Slack webhook is the production pattern). Restrict who may respond with assigned_users — a list of {id, name} pairs — so only named approvers can act on the gate, and the approver field in the audit trail is always a real person.

# Slack side (incoming webhook app):
# 1. approval task defers -> webhook posts to #data-oncall
#    "Approve schema change on prod.orders? (sarah.d on call)"
# 2. 30 min later, no response -> workflow posts to #data-escalation
#    "Still waiting on approval — 30 min left before timeout"
# 3. approve/deny button click -> webhook back to Airflow
#    -> task resumes with event {approved: true, approver: 'sarah.d', comment: '...'}
#
# Airflow side: HITLOperator(notification_webhook_url=..., timeout=3600*6)
# Approval decisions log to metadata DB with timestamps.

4. Timeout and Expiry of Approval Tasks

An approval that waits forever is a pipeline outage in slow motion. The HITLOperator timeout bounds the wait: on expiry the gate fails, downstream tasks don't run, and the failure is visible — loudly, with an alert path if you wired notifications correctly. That's the design intent: stale approvals become loud failures, not silent stalls.

The timeout should match the operation's SLA, not your optimism. An overnight migration gate can wait six hours; a prod deploy gate should force a decision in minutes. When the timeout and the SLA disagree, the gate is lying to the business about when decisions happen.

Two failure paths need planning. Denied: the DAG fails with a clear message and an audit record — decide whether a retry is allowed or the run is abandoned. Timed out: the pipeline failed without a human decision, which is the outcome you designed for; the runbook should say who gets paged and what the recovery rerun looks like.

Timeout doesn't have to mean failure: set defaults to one of the options and the timer applies it automatically — an approval gate can auto-reject (defaults=["Reject"]) or auto-approve at expiry, converting a missed decision into a defined outcome instead of a failure. The timeout argument is response_timeout on Airflow 3.3+ (3.1/3.2 used execution_timeout), and since 3.3 the scheduler's response-timeout sweep enforces it even while no triggerer is held.

5. Where HITL Fits: Destructive DDL, Prod Deploys, Money Movement

HITL earns its keep on the operations where a wrong answer is expensive and a right answer requires judgment. Destructive DDL — drops, truncates, renames — tops the list: the schema change that skipped review in this article's incident is the canonical case. Production deploys run a close second, and money movement third: payments, refunds, batch settlements where an unattended error is a financial incident.

The line is reversibility. If the operation can be undone cheaply and safely, automation wins — a gate there is friction without protection. If the operation is one-way, gate it. If the operation is one-way and expensive to undo, gate it with dual approval and a longer review window.

The pattern composes. A deploy pipeline can carry several gates: one before the migration, one before the cutover, one before the rollback decision. Each gate is cheap to run — it defers, not blocks — so the cost is human attention, which is exactly the resource that should be spent.

6. HITL vs Sensors vs External Waits

Three tools pause a pipeline, and teams routinely use the wrong one. A sensor waits for a condition in the world — a file, an API, a partition — and resumes automatically when it appears. HITL waits for a human decision, which can't be detected by polling. ExternalTaskSensor waits for another DAG's task to finish, a pure scheduling dependency.

The test is what completes the wait. If a machine can check it, it's a sensor. If a person must decide, it's HITL. If it's about ordering between DAGs, it's a cross-DAG dependency. Mixing them up creates two failure classes: pipelines that stall waiting for a human when a file check would do, and pipelines that skip the human entirely because someone modeled the approval as a sensor condition.

7. Designing the Approval Contract

A gate is only as good as the contract around it. Write it down before the DAG: who approves (named role, not 'the team'), what they approve (exact scope of the change), how long they have (timeout tied to SLA), what happens on deny, and where the decision is recorded. The contract is what makes the gate auditable and the approver accountable.

Operationally, three things keep the contract honest. First, the gate must be structurally unskippable — no trigger rule that bypasses it, no conditional branch around it, no 'approve if the ticket exists' shortcut. Second, decisions log identity and comment, because post-incident reviews are won or lost on that record. Third, test the gate's failure modes in staging: deny, timeout, and no-notification, so the failure path is proven before the first real deployment.

This is the difference between review theater and review. The schema change that skipped review happened because the contract was implicit. Make it explicit, and the gate stops being a formality and starts being a control.