Airflow Configuration: One Typo That Took Down Workers

One duplicated section header took down every worker. The file looked right at a glance and parsed wrong on every restart.

Config feels boring until it's the blast radius. A single typo in airflow.cfg fans out to schedulers, workers, and webservers simultaneously.

Treat config as code with validation gates. You'll learn that discipline here.

The Config Hierarchy Explained

Effective config resolves in seven layers: plain env var first, then _CMD env, then _SECRET env, then airflow.cfg value, then _cmd in file, then _secret in file, and built-in defaults last. Each layer overrides the next, which means a stray env var beats a careful file edit every time. Only nine sensitive keys support _cmd/_secret derivation: database sql_alchemy_conn, core fernet_key, celery broker_url/flower_basic_auth/result_backend, atlas password, smtp smtp_password, api secret_key, and api_auth jwt_secret.

That layering is why debugging starts with airflow config list, not with reading the file. The list command prints what services actually use, including which source won; add --include-sources, --include-env-vars, or --section core to slice it. Get one key fast with airflow config get-value core executor. Files show intent; list shows truth.

Design for it: files carry dev defaults, env carries deployed overrides, backends carry secrets. Never the reverse. For security-sensitive fleets, scope env vars to the components that need them instead of sharing every secret everywhere, and keep clocks in sync with ntpd or log and API auth calls start failing.

Env-Var Precedence in Practice

Env vars follow AIRFLOW__SECTION__KEY with double underscores: AIRFLOW__CORE__PARALLELISM maps to [core] parallelism. Uppercase throughout, dots become underscores, so [providers.some_provider] this_param becomes AIRFLOW__PROVIDERS_SOME_PROVIDER__THIS_PARAM. One typo in the section name creates a silently ignored variable. The same naming extends to _CMD and _SECRET variants for the nine derivable keys.

Prefer env for anything that differs per environment: executors, parallelism, DB URLs, heartbeat intervals. Generate a commented baseline with airflow config list --defaults > "$AIRFLOW_HOME/airflow.cfg", then uncomment only what you change so upgrades inherit new defaults. Keep airflow.cfg minimal so diffs stay readable.

Audit env regularly. Stale exports in systemd units and compose files outlive the engineers who set them and override current intent.

# deployed overrides (single source per env)
export AIRFLOW__CORE__EXECUTOR="CeleryExecutor"
export AIRFLOW__CORE__PARALLELISM="64"
export AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC="10"
export AIRFLOW__DATABASE__SQL_ALCHEMY_CONN="postgresql+psycopg2://airflow:${DB_PASS}@pg:5432/airflow"

# verify what services actually see
airflow config list | grep -iE 'executor|parallelism|heartbeat'
airflow config lint  # catches renamed/removed keys after upgrades

Config-as-Code in CI

Config-as-code means every environment's effective config lives in version control and deploys through CI. Env files per environment, Helm values for clusters, secrets referenced never embedded. Diffs between staging and prod show exactly what differs.

Gate changes: airflow config lint catches renamed keys, airflow config list snapshots catch drift, DAG parse tests catch fallout. A config PR that fails lint never reaches workers.

Canary the rollout: one scheduler first, watch heartbeats and queued age, then the fleet. Config errors announce themselves within minutes when you watch the right signals.

Validating With Config List

airflow config list prints effective values with origins; run it on broken and healthy hosts and diff. airflow config lint flags removed or renamed keys after upgrades. Both belong in CI and in runbooks.

The duplicate-section check is one grep: grep -n '^\[' airflow.cfg should show each header once. Twice means the parser silently picks a winner and your workers run on values you didn't choose.

Snapshot effective config nightly. A cron writing airflow config list to versioned storage turns mystery drift into a dated diff.

[core]
parallelism = 64
max_active_tasks_per_dag = 8

[scheduler]
scheduler_heartbeat_sec = 10
dag_dir_list_interval = 60

# check for the killer: duplicated headers
# grep -n '^\[' airflow.cfg
# [core] must appear exactly once

Plugins and the Plugin Folder

Some config isn't INI at all. Cluster policies, advanced logging, DAG serialization, pod mutation hooks, parse timeouts, UI customization, extra exported variables, and DB setup live in airflow_local_settings.py. Put that file in $AIRFLOW_HOME/config (on sys.path at init), never in dags/: since 2.10.1 the dags folder left sys.path, so settings parked there silently never import.

Guard every plugin import with try/except and lazy loading. Parse-test plugins in CI with airflow dags list-import-errors. Version the folder alongside DAGs so rollbacks revert both together.

Keep plugins small. Business logic belongs in tasks and packages, not in startup-loaded modules that can halt the fleet.

Version-Specific Config Drift

Upgrades rename and remove keys regularly. Old files carry dead names that new code ignores, so services run on defaults while operators believe their tuning applies. That's how parallelism silently halves overnight.

The defense is mechanical: pin versions, read the changelog's config section, run lint, diff effective config staging versus prod. Ten minutes of checklist saves a night of worker triage.

Track renames in your config repo's history. Future you will thank past you when the next major bumps the same keys.