Airflow Providers: The Upgrade That Renamed Our Operator

Your DAG imports PostgresOperator at the top of the file. It's been there for months, and it's worked every night. Then someone rebuilds the image, bumps a few packages, and suddenly the scheduler can't parse a single DAG. The operator didn't disappear — it moved.

That's the provider economy in one scene. Airflow stopped being a monolith; the integration layer is now 100+ pip packages, each versioned and released on its own cadence. Power comes with a new failure mode: upgrade risk you never used to have.

This article maps the provider landscape, shows the three layers you'll actually touch, and gives you the pinning discipline that makes upgrades boring again. Because a scheduler that can't parse DAGs is a company that can't load data — and that's not a question of if, it's when.

1. What Providers Are (and Why 100+ Exist)

A provider is a pip package like apache-airflow-providers-postgres. It ships operators, hooks, sensors, and transfer operators for one external system, with its own version number and release cadence. Airflow core stays lean; the integrations plug in as wheels.

Why split them? The Postgres community releases at its own pace, the AWS provider tracks SDK changes on its own, and nobody wants an S3 SDK bump to force a core upgrade. Decoupled release trains, same platform.

The cost is what you just survived: 100+ packages, each a potential drift point. The response isn't to avoid providers — it's to treat them as dependencies with discipline, the same way you treat any other library.

Installation is an extra away: apache-airflow[postgres,amazon] installs the matching provider packages together with Airflow under the published constraints, and once a provider is installed, Airflow discovers its operators, hooks, and sensors automatically after the next restart. Each provider follows SemVer — backward compatible within a major version — and any breaking change, like a renamed operator, is documented in that provider's release notes.

2. The Provider Catalog for the Modern Data Stack

The catalog matches the stack you actually run: database providers (postgres, mysql, sqlite, snowflake), cloud providers (amazon, google, azure), and transfer operators that move data between systems in one task — S3ToSnowflakeOperator being the classic.

Each provider is a toolbox: Postgres gives you PostgresOperator for SQL, PostgresHook for connections, and PostgresToS3Operator for transfers. You rarely need more than one or two per system.

The rule for choosing: if the system has a provider, use its operator for the standard step and its hook for custom logic. Only drop to raw SDK code when the provider is missing a feature — and file the feature request while you're at it.

One registry to search: the Airflow Registry indexes every provider's operators and hooks (300+ hooks at last count), and the Operators and Hooks Reference lists them all on a single page. Every hook descends from BaseHook, which is why connections behave the same way across all of them.

3. Operators vs Hooks vs Plain Python

Three layers, three jobs. The operator is the atomic step: run this SQL, copy this file, wait for this condition. The hook is the connection client: it owns credentials, hosts, and how to connect — and any task can call it. Plain Python is everything bespoke that neither covers.

The practical split: a standard step like "run this statement" gets an operator. A custom sequence that needs a connection gets a hook inside a Python task. Pure logic with no external system gets a plain @task.

This split keeps your DAGs declarative at the top and honest in the middle. And it means the provider upgrade risk concentrates where it's visible: one import per operator, one hook call per connection.

The ecosystem's rule of thumb: when an operator already wraps a hook for your use case, prefer the operator; reach for the hook directly only for logic an operator can't express. And if you write your own operator, it should call a hook internally rather than talk to the external API itself.

from datetime import datetime, timedelta

from airflow.decorators import dag, task
from airflow.providers.postgres.hooks.postgres import PostgresHook
from airflow.providers.postgres.operators.postgres import PostgresOperator


@dag(
    schedule="@daily",
    start_date=datetime(2026, 7, 1),
    catchup=False,
    default_args={"retries": 2, "retry_delay": timedelta(minutes=5)},
)
def orders_report():
    # Layer 1: operator wraps the whole SQL step
    create_report = PostgresOperator(
        task_id="create_report_table",
        postgres_conn_id="warehouse_conn",
        sql="""
            CREATE TABLE IF NOT EXISTS daily_orders AS
            SELECT order_date, COUNT(*) AS orders FROM orders GROUP BY order_date
        """,
    )

    # Layer 2: hook inside a task for custom logic
    @task
    def validate_row_count() -> int:
        hook = PostgresHook(postgres_conn_id="warehouse_conn")
        count = hook.get_first("SELECT COUNT(*) FROM daily_orders")[0]
        if count == 0:
            raise ValueError("Report table is empty")
        return count

    # Layer 3: plain Python, no connection at all
    @task
    def summarize(count: int) -> dict:
        return {"rows": count, "expected_floor": 1000}

    create_report >> validate_row_count() >> summarize()


orders_report()

4. The Import-Time Silent Failure

The incident pattern is always the same: an import at module top level, a provider bump that moves the symbol, and the scheduler's parse loop hitting ImportError on every DAG file. Broken imports don't fail a task — they unload the whole DAG.

Why does top-level import make it worse? The scheduler imports your file on every parse loop. A guarded import inside the function can at least isolate the damage; a top-level import kills parsing before Airflow ever sees your tasks.

Two mitigations: import inside functions when the module is heavy or version-fragile, and keep a parse-time guard that logs a clear message instead of a raw traceback.

try:
    from airflow.providers.postgres.operators.postgres import PostgresOperator
except ImportError:
    raise RuntimeError(
        "apache-airflow-providers-postgres is missing or outdated. "
        "Install the pinned version from requirements.txt and rebuild."
    ) from None


# Alternative for version-fragile providers: import inside the callable
from airflow.decorators import task


@task
def run_validation():
    from airflow.providers.postgres.hooks.postgres import PostgresHook  # lazy import

    hook = PostgresHook(postgres_conn_id="warehouse_conn")
    return hook.get_first("SELECT 1")

5. Pinning and Upgrade Discipline

The fix that ends this incident class: pin exact versions, freeze the lockfile, and audit before and after every bump. requirements.txt becomes the single source of truth for provider versions across dev, CI, and prod.

The upgrade path is a checklist, not a hope: read the changelog for renamed or moved operators, bump in staging, run airflow providers check plus a full DagBag parse, and only then roll to prod.

The CLI completes the loop. airflow providers list shows every installed provider and version; airflow providers check validates the current installation against your constraint. Thirty seconds of audit every week beats a 5 AM call every quarter.

The CLI family is wider than two commands: airflow providers get <provider_name> prints a single provider's details, and airflow providers hooks and airflow providers secrets list what each provider contributes — a five-second pre-bump audit of the whole fleet.

apache-airflow==3.3.0
apache-airflow-providers-postgres==5.18.1
apache-airflow-providers-snowflake==5.12.0
apache-airflow-providers-amazon==8.21.0
apache-airflow-providers-sqlite==3.9.0

6. Building Your Own Provider Structure

Sometimes the catalog isn't enough — an internal system needs an operator, or the team wants consistent wrappers. You don't need a published package; Airflow discovers providers from a folder on the PYTHONPATH or, for Airflow 3, from plugins.

The layout mirrors the official structure: an operator package, a hooks package, and an __init__ exposing them. The operator itself subclasses BaseOperator, defines template_fields, and calls its hook from execute. Keep it small — one or two operators that encode team standards beat a sprawling internal framework.

Version your internal provider like the official ones: semantic versions, a changelog, and the same pinning rules. The moment it's consumable by other DAGs, it's infrastructure — and infrastructure deserves release discipline.

The official docs include a full "How to create your own provider" guide, and the providers you build have exactly the same capabilities as community ones — same auto-discovery, same plugin points — so an internal provider can be released and shared like an official one.

from airflow.models import BaseOperator
from airflow.utils.context import Context
from my_analytics_provider.hooks.reporting import ReportingHook


class DailyReportOperator(BaseOperator):
    template_fields = ("report_name",)

    def __init__(self, report_name: str, **kwargs) -> None:
        super().__init__(**kwargs)
        self.report_name = report_name

    def execute(self, context: Context) -> str:
        hook = ReportingHook()
        return hook.build_daily(self.report_name, context["ds"])