Airflow Variables & Pools: The 10,000-Line JSON Mistake

The scheduler reads your DAG files on a loop, hundreds of times a day. Every parse should be cheap. Ours wasn't — because someone parked a 10,000-line JSON inside an Airflow Variable, and every DAG file in the folder paid for it.

Variables are the most abused feature in Airflow. They're meant for small config flags: a region code, a toggle, a default bucket name. Used right, they're invisible. Used as a config store for a big mapping file, they tax every parse of every DAG until scheduling visibly slows.

Pools are the other half of this article: the concurrency gates that keep 30 tasks from smashing the same database at once. Together, variables and pools are how you keep config small and load controlled.

If your Variables screen has a config_catalog entry with a scrollbar, your parse loop is paying rent for it. This article is the eviction notice.

1. Variables: When They Help, When They Hurt

A Variable is a key/value row in the metadata DB, created via the UI, the CLI (airflow variables set), or an environment variable (AIRFLOW_VAR_REGION_CODES). Perfect for small config flags: a region code, a feature toggle, a default bucket name, a retry threshold.

When they hurt: treated as a config store. Giant JSON mappings, secrets (variables get no real protection — values are visible in the UI and API even when Fernet-encrypted at rest), and values read at module level in DAG files.

The size rule is brutal and simple: if you can't read the value in one glance, it doesn't belong in a Variable.

Two clarifications from the docs. Values stored in the metadata DB are encrypted at rest with the Fernet key, but that encryption doesn't hide them — they render in plaintext in the UI and API, which is why the no-secrets rule still stands. And variables set via AIRFLOW_VAR_* never appear in the UI or airflow variables list at all; they're resolved at runtime on the worker process, which is exactly why env-var precedence can differ between environments.

2. The Parse-Time Fetch Trap

The scheduler parses every DAG file on every loop, in every worker process. Anything at module level runs each time — and that includes Variable.get. A single module-level read turns your parse loop into a DB round-trip treadmill.

The fix is placement: fetch inside the task, where it runs once per task execution, not once per parse. The value is read at task runtime, so it's also fresher — a UI edit picks it up without waiting for the next parse.

Our incident DAG fetched config_catalog at module level. Moving one line from module scope into the task callable took parse time back from over a second to ~300ms.

The docs endorse two escape hatches when a top-level read is unavoidable. Reference the variable through Jinja templating ({{ var.value.config_catalog }}), which delays the fetch until task execution, or set AIRFLOW__SECRETS__USE_CACHE=True (Airflow 2.7+) to cache Variable.get results during DAG parsing with a 15-minute TTL.

from airflow.models import Variable

# BAD: runs on every scheduler parse of this file
CATALOG = Variable.get("config_catalog")


from airflow.decorators import dag, task
from pendulum import datetime


@dag(schedule="@daily", start_date=datetime(2026, 8, 1), catchup=False, tags=["orders"])
def daily_orders():
    @task
    def load_orders():
        # GOOD: runs once per task execution, not once per parse
        catalog = Variable.get("config_catalog")
        ...

    load_orders()

3. Large Configs Belong in Files or Object Storage

A 10,000-line catalog is not a value. It's a document — versioned, reviewed, rollback-able, and readable by the whole team. Put it in git or object storage and let the DAG reference it by URI.

The parse-time cost drops to one tiny read: a URI string. The payload loads at task runtime, when it's actually needed, and changes to the catalog never touch the scheduler loop.

Version the file, review changes, and roll it back like any other artifact. That's what a config store gives you — and what a Variable row never will.

import json

from airflow.decorators import dag, task
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
from pendulum import datetime

CONFIG_URI = "s3://configs/analytics/catalog.json"


@dag(schedule="@daily", start_date=datetime(2026, 8, 1), catchup=False, tags=["config"])
def catalog_loader():
    @task
    def load_catalog() -> dict:
        hook = S3Hook(aws_conn_id="aws_analytics")
        body = hook.read_key("analytics/catalog.json", bucket_name="configs")
        return json.loads(body)

    load_catalog()

4. Pools: Capping Concurrent Tasks

A Pool is a named slot budget shared across all DAGs. Tasks assigned to the pool can't run more instances than the pool has slots — even when the executor has capacity. It's the ticket counter that stops a thundering herd on your warehouse or API.

Create pools with the CLI: airflow pools set warehouse_load 4 "warehouse quota". Assign tasks with pool="warehouse_load" in default_args or per task. Tasks without a pool draw from default_pool, bounded by executor parallelism.

Our fix used two pools: one for warehouse loads, one for API-bound calls. Same fleet, same schedules — but the warehouse stopped getting slammed by 30 concurrent loads at midnight.

Three pool facts from the docs. default_pool starts with 128 slots and cannot be removed. A task can occupy more than one slot via pool_slots when it's computationally heavy relative to its pool-mates. And a task assigned to a pool that doesn't exist simply never runs — Airflow raises no error, so validate pool names in CI.

from airflow.decorators import dag, task
from pendulum import datetime


@dag(
    schedule="@daily",
    start_date=datetime(2026, 8, 1),
    catchup=False,
    default_args={"pool": "warehouse_load", "retries": 2},
    tags=["billing"],
)
def daily_billing_rollup():
    @task
    def rollup_billing():
        ...

    rollup_billing()

# create the pool once:
# airflow pools set warehouse_load 4 "warehouse quota"

5. priority_weight and Starvation

When a pool is full, priority_weight decides who gets the next free slot. Default weight is 1, and scheduling is weighted round-robin — so a 50-weight dashboard task gets the slot before a pile of weight-1 batch backfills.

Starvation is real: if batch jobs and the nightly dashboard share one pool at equal weight, the dashboard waits like everything else. That's a design decision wearing a bug costume.

Set weights deliberately: 50 for the dashboard load, 1 for batch backfills. Or split the pool so production data paths never compete with research queries.

6. Queues as Worker Routing

The queue attribute is routing, not capacity: queue="heavy_etl" sends the task to workers listening on that queue (Celery: worker --queues=heavy_etl). Pools cap how many; queues pick which machines run them.

Use queues to isolate workloads by resource profile: memory-hungry transforms on big workers, API calls on network-isolated ones. Pools and queues compose — a task can be both queue-routed and pool-capped.

The confusion trap: a task routed to a queue with no listening workers sits queued forever, looking exactly like a pool bottleneck. Check the worker's queues before you blame the pool.

from airflow.operators.python import PythonOperator
from pendulum import datetime


with DAG(
    dag_id="feature_build",
    schedule="@daily",
    start_date=datetime(2026, 8, 1),
    catchup=False,
    tags=["ml"],
) as dag:
    build_feature_matrix = PythonOperator(
        task_id="build_feature_matrix",
        python_callable=lambda: print("matrix built"),
        queue="heavy_etl",  # routed to workers with big memory profiles
    )

7. The Parse Loop That Stays Fast

The postmortem produced one standing rule: parse time is a budget, and Variables are a cost line. Inventory quarterly — airflow variables list, then size the big ones. Anything over a few KB moves to files or object storage.

Watch the scheduler heartbeat lag after every deploy. A DAG that grew a module-level read will show up as a parse-time jump before it ever shows up as a red task.

And run the concurrency side by the same discipline: every shared system gets a pool, every pool has an owner, and weights are reviewed when contention appears. Config small, load capped, loop fast.