Airflow DAGs: The Import Time Trap That Stalls Scheduling

Your DAG file has two lives. When you run it by hand, it's a normal Python script. When the scheduler runs it, it's parsed on every loop — and your "normal script" runs inside the scheduler's critical path. That split is where the import time trap lives.

A team learned this the hard way when one DAG file opened a database connection at module top level. Parse time for that file jumped from ~300ms to 12 seconds, and because the scheduler parses every file on every loop, one lazy connection slowed the entire deployment's scheduling.

This article dissects the DAG file: anatomy, default_args, discoverability, the parse loop, and the purity rule that keeps imports fast. The incident runs through the middle of it, because the trap is invisible until it isn't.

A DAG file that does work at import time is a landmine with a schedule.

1. Anatomy of a DAG File

A DAG file is Python that imports, builds DAG objects, and does nothing else. The scheduler collects every DAG object it can import from the dags_folder and registers them. One file, one DAG is the convention — multiple DAG objects per file parse fine, but review gets harder.

The modern skeleton: an @dag-decorated function, default_args at the top, tasks inside, and dependencies wired at the end. Every DAG gets a dag_id (unique, stable), a schedule, a start_date, and tags.

The file is declarative on purpose. It describes the graph; it should never execute the work it describes.

Two loading details are worth knowing. By default the loader only parses files whose content contains the strings airflow and dag (case-insensitive) — a DAG file that matches neither string can quietly never register. And an .airflowignore file (gitignore-style patterns) excludes matching files from parsing entirely.

from datetime import datetime

from airflow.decorators import dag, task


default_args = {
    "owner": "data-platform",
    "retries": 3,
    "retry_delay": 300,
}


@dag(
    dag_id="market_etl",
    schedule="@daily",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    default_args=default_args,
    tags=["etl", "market-data"],
)
def market_etl():
    @task
    def extract_stock_data() -> list[dict]:
        return [{"ticker": "AAPL", "close": 212.4}]

    @task
    def load_prices(rows: list[dict]) -> None:
        print(f"loading {len(rows)} rows")

    load_prices(extract_stock_data())


market_etl()

2. default_args and Why Inheritance Surprises People

default_args is a dict the DAG merges into every task — retries, retry_delay, owner, email settings. It's a convenience, and it's also a shared reference. Airflow doesn't deep-copy your dict; tasks inherit values by reference.

The trap: mutable values. Put a list or dict inside default_args and every task sees the same object. One task mutating it changes what the next task inherits. Strings, ints, and tuples are safe; containers are not.

The rule: keep default_args immutable and read-only. If a value must differ per task, set it on the task, not in the defaults.

3. tags, doc_md, and Discoverability

A 500-file DAG repository is a library. tags, description, and doc_md are the catalog: tags filter the UI's DAG list, description shows in the DAGs table, doc_md renders as rich documentation in the Docs menu.

Cheap to write, expensive to retrofit. A DAG missing tags is a DAG nobody finds. A DAG with doc_md explaining its trigger, owner, and failure procedure survives its author leaving the company.

Write doc_md as the runbook: what it loads, which connection it uses, what to do when it fails. The Docs menu renders it — make it a habit, not a chore.

4. The Parse Loop: What the Scheduler Reads and When

The scheduler runs a loop: re-parse the dags_folder, diff the results against what it knew, schedule any DAG whose next data interval has arrived. This loop is the heartbeat of the whole platform — every file gets imported, every time, forever.

That's why parse duration is a production metric. Default parses hover around 300ms per file. The scheduler uses process pools and caching, but the loop is serialized per file: one 12-second file delays the entire cycle.

The output of a parse is a DAG object the scheduler can diff — which is why parse must be pure. If a parse depends on the time, the weather, or the state of a database, the diff is unreliable and the schedule drifts.

Under the hood two processes do the work: the DagFileProcessorManager, an infinite loop that decides which files need parsing (it runs standalone via the airflow dag-processor CLI command), and one DagFileProcessorProcess per file that converts it into DAG objects. Three config knobs govern the loop: min_file_process_interval (seconds between re-parses — lower means fresher DAGs but more CPU), parsing_processes (how many files parse in parallel), and dagbag_import_timeout — a hard ceiling on the module import, so a file that hangs is reported as an import error instead of stalling the loop forever.

5. Top-Level Code Must Be Side-Effect Free

The purity rule: module scope may define, not execute. Constants, imports, DAG objects, default_args — fine. Opening a DB connection, calling an API, writing files, or running heavy computation at import — a production incident waiting to happen.

Why so strict? Because the scheduler imports your file in a process you don't control, on a loop you can't pause, alongside files you don't own. Your side effect isn't just your problem — it delays the whole parse cycle, every cycle.

The incident in this article was exactly this: a top-level get_conn() call turned 300ms into 12 seconds. The fix was lazy: import the client and open the connection inside the task callable, where they belong.

Two more top-level traps the docs call out. Imports count as top-level code: an import that executes heavy work at module scope slows every parse of that file. And Variable.get() at module scope issues a metadata DB request on every parse unless caching is enabled — prefer Jinja templates ({{ var.value.name }}) inside task code, or cache the lookup.

# The trap: a connection opened at import time
# from market_db import get_conn
# get_conn()  # 12 seconds, every parse, every loop

# The fix: pure module scope, lazy work inside tasks
from datetime import datetime
from airflow.decorators import dag, task


@dag(dag_id="market_etl", schedule="@daily", start_date=datetime(2026, 1, 1))
def market_etl():
    @task
    def load_market_prices() -> None:
        from market_db import get_conn  # lazy import: parse stays pure
        conn = get_conn()
        conn.execute("INSERT INTO prices SELECT * FROM staging")
        conn.close()

    load_market_prices()


market_etl()

6. The @dag Decorator and Modern Style

The @dag decorator turns a function into a DAG and turns @task functions into tasks wired by return values. No with-block, no manual xcom_push — TaskFlow-style signatures carry data between tasks.

The classic with DAG(...) as dag: style still appears in legacy codebases and stays valid; Airflow 3 maintains it. New code should use the decorator: it's the official tutorial style, it type-checks better, and it hides the XCom plumbing.

One caveat: decorators don't exempt you from the purity rule. The decorator function itself still runs at import time. Keep the body declarative; keep imports inside the task callables.