Airflow Configuration: One Typo That Took Down Workers

Every Airflow team has the same file, and almost nobody reads it twice. airflow.cfg is 300 lines of settings that look harmless — until one duplicated section, one wrong key name, or one version-gated option takes the whole fleet down at 3 AM. You'll spend more time debugging config than debugging DAGs, so treat config like code.

The config system is a chain of four layers: built-in defaults, airflow.cfg, environment variables, and secrets backends. Each layer silently overrides the one below it. That's powerful — and it's exactly why a typo can hide: your file says one thing, your container env says another, and nobody bothered to ask who actually won.

The painful truth: there's no such thing as "one config." There's dev, staging, prod, and three schedulers that all disagree. The teams that survive manage config from a single source per environment and validate before deploying. The teams that don't learn at 3 AM.

This one's about winning that fight before the phone rings.

1. The Config Hierarchy: Defaults → File → Env → Secrets

Airflow resolves every setting through four layers. Built-in defaults ship with the code. airflow.cfg in AIRFLOW_HOME overrides them. Environment variables override the file. And a secrets backend can override any of it at runtime.

Each layer wins over the one below it, wholesale. If the file defines 40 keys and the env defines 1, that 1 env key wins and the other 39 still come from the file. No merging, no warnings — the winner is just whoever is higher.

The trap: your container image may embed an airflow.cfg, your deployment manifest may inject env vars, and your CI may render yet another file. Three sources, one effective result. If you review only one of them, you're reviewing a lie.

Two env-var variants sit between the plain env var and the file in the official precedence order: AIRFLOW__SECTION__KEY_CMD derives the value by running a command, and AIRFLOW__SECTION__KEY_SECRET fetches it from a secrets backend. Both rank above airflow.cfg but below the plain env var, and both are supported for a short list of keys — sql_alchemy_conn, fernet_key, broker_url, result_backend, and smtp_password among them. The full order of resolution: env var, env _CMD, env _SECRET, file value, file command, file secret, built-in default.

Production teams treat the effective config, not the file, as the artifact that matters.

# The same setting, three layers — env always wins
airflow config get-value core parallelism
# 32  (from built-in default)

# 1. default:           core.parallelism = 32
# 2. airflow.cfg:       [core] parallelism = 16
# 3. env var:           AIRFLOW__CORE__PARALLELISM=8

airflow config get-value core parallelism
# 8   (env var layer won)

2. Env-Var Precedence and the AIRFLOW__SECTION__KEY Pattern

The env-var naming scheme is the whole game: AIRFLOW__, then the section in uppercase, then the key, all separated by double underscores. AIRFLOW__CORE__PARALLELISM maps to [core] parallelism. AIRFLOW__CELERY__WORKER_CONCURRENCY maps to [celery] worker_concurrency.

Because env vars never touch the file, they're the standard lever for containers, systemd units, and CI. One image, many environments — the env renders the difference at process start.

Two gotchas. First, env vars are read when the process starts, not per task — changing them on a running scheduler does nothing until restart. Second, they must be set for every component that needs them: a scheduler with the var and a worker without it will disagree all night.

Two more rules from the docs. Dotted section names translate to underscores: [providers.some_provider] this_param becomes AIRFLOW__PROVIDERS_SOME_PROVIDER__THIS_PARAM — the dot is illegal in the env-var form. And the AIRFLOW_CONFIG env var overrides the config file path itself, which is how containers point every component at one rendered file. Sharing rules differ by key: the JWT signing key ([api_auth] jwt_secret) must be identical on every component that generates or validates tokens, while database connection strings and Fernet keys should be scoped only to the components that need them.

Treat the env block as the deployable config artifact. Version it, review it, validate it — exactly like code.

# The pattern: AIRFLOW__SECTION__KEY
AIRFLOW__CORE__PARALLELISM=32
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG=8
AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT=60
AIRFLOW__SCHEDULER__PARSING_PROCESSES=4
AIRFLOW__CELERY__WORKER_CONCURRENCY=8
AIRFLOW__WEBSERVER__WEB_SERVER_WORKER_TIMEOUT=120

export AIRFLOW__CORE__PARALLELISM AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG

# Verify what the process will actually see before you start it:
airflow config get-value core parallelism
# 32

3. Config-as-Code in CI: Render, Validate, Diff

Config-as-code means the env block and airflow.cfg live in the repo, get reviewed in pull requests, and get validated before they touch a host. The validation is the entire point — a typo caught in CI costs 30 seconds, the same typo in prod costs the night.

Render from a template per environment, then prove the render is sane. The proof is a three-command pipeline: render, diff against the previous version, and read back the effective values with airflow config list on the real image.

The read-back is the part teams skip. A rendered file that parses cleanly can still override the wrong section, point at the wrong database, or silently reset a knob — because the merge with defaults happens at runtime, not at render time.

Gate the deploy on the diff. If a rollback-shaped diff lands in the pull request, it should need a human to approve it before it ships.

4. Validating With airflow config list

airflow config list is the swiss-army tool of config debugging. Run it and you get every setting, its source, and whether it came from defaults, the file, or an env var — which is exactly how you catch the sticky-note war between layers.

The output marks each option with where it came from, so the command is also your documentation: it tells you which keys exist in your installed version. That matters more than you'd think — Airflow 2.x and 3.x renamed and dropped settings between releases.

Pair it with airflow config get-value for surgical checks: one command, one answer, the winner of the whole stack. It's the fastest way to answer "why is parallelism 32 when my file says 16?"

One flag to know: airflow config list --defaults prints the built-in defaults, which is the sanctioned way to generate a fresh, clean airflow.cfg from scratch instead of inheriting one across versions and upgrades.

Build it into the runbook. Every config-related incident starts with the same two commands — you want them to be muscle memory.

# See every setting, with its source layer
airflow config list

# Answer one question: what does parallelism actually resolve to?
airflow config get-value core parallelism
# 32

# Prove which layer is winning (file vs env) for a suspicious key
airflow config get-value scheduler scheduler_heartbeat_sec
# 5

# Spot-check the whole stack against your baseline in one shot
diff <(airflow config list) baseline/effective.txt && echo "config OK"

5. Plugins and the Plugin Folder

Plugins extend Airflow — custom operators, hooks, sensors, even whole UI views — and they're loaded from plugins_folder, a config key in its own right. Misset it and Airflow runs fine, just without half your custom code.

The folder is read at process start. A plugin with a broken import fails the load of every plugin in that folder, and the failure mode is rarely a crash — it's a silent absence. Your custom operator simply isn't there when the DAG tries to import it.

Keep plugins versioned like DAGs: pinned versions, changelog review, and a smoke test that imports every plugin module before deploy. A plugin that fails to import in CI is a 30-second fix; the same failure in prod is a blocked schedule.

6. Version-Specific Config Drift

Config is versioned too — silently. Upgrading Airflow from 2.8 to 3.x renamed settings, moved sections, and dropped keys outright. A 2.x airflow.cfg copied forward either fails validation or, worse, quietly ignores the parts that no longer exist.

Drift is dangerous because it's invisible: the config parses, the defaults kick in, and you run months on settings you never chose. Our duplicate-section incident was one form; version drift is the cousin that arrives with every upgrade.

The defense is a baseline. Capture airflow config list from the current version, store it with the release, and diff it after every upgrade. If a tuned knob drops out of the effective config, the diff will scream about it.

Read the changelog section on configuration for every minor bump. Airflow documents renamed keys — the teams that skip this page are the teams that file the incident tickets.

# After any Airflow upgrade, before pointing traffic at it:
airflow version
# 3.3.0

airflow config list > baseline/after-upgrade-3.3.txt

diff baseline/prod-2.8.txt baseline/after-upgrade-3.3.txt
# Only the settings YOU care about should appear.
# Any tuned knob that vanished from the output is now a default.

7. Incident Deep Dive: Failure Flow → Fix

Replay the night. The config push landed at 01:00. It contained a duplicated [core] section — the merge of two feature branches re-added a block that already existed. The parser read the file, took the second block as authoritative, and reset parallelism, executor, and max_active_tasks to their built-in defaults.

At 01:00 the new workers came up, read the reset values, and crashed on startup — mismatched executor settings, one after another, in a loop. The webserver never noticed, because the webserver doesn't run tasks. The scheduler looked alive, because the scheduler process was alive. The fleet was down and the only loud signal was the absence of DAG runs.

The fix had three parts. First, a validated render: the effective config is now produced in CI, diffed against the live baseline, and gated before any host restarts. Second, env vars replaced file edits for every environment-specific knob — the file is now read-only on the servers. Third, the incident runbook starts with airflow config list, because the resolved value, not the pushed file, is what broke the night.