Airflow on Kubernetes: 400 Pods, 3 Nodes, 1 Nightmare

Four hundred pods met three nodes on a Tuesday night. The cluster didn't degrade gracefully; it fell over sideways.

Every task had launched a pod with no resource requests and no limits. Kubernetes packed them wherever they fit until nothing fit anywhere.

Pod-per-task is powerful precisely because it's expensive. You'll learn to price that power here.

KubernetesExecutor: A Pod Per Task

KubernetesExecutor trades warm reuse for clean rooms. Every task instance becomes a pod: fresh filesystem, declared image, explicit env, bounded resources. When it finishes, the pod dies and takes its mess with it.

That isolation kills entire failure classes: dependency conflicts between tasks, leaked /tmp state, noisy neighbors hogging RAM. It replaces them with a new cost: 20-60 seconds of pod startup per task.

Use it where isolation pays: untrusted code, conflicting dependencies, spiky resource needs. Avoid it where it taxes: thousands of two-minute tasks that warm Celery workers finish faster.

Helm Install of the Official Chart

Install the official chart with a pinned version, a private registry mirror, and values in Git. Chart versions drift fast; floating tags turn routine deploys into template-rename incidents. One-command install looks like helm repo add apache-airflow https://airflow.apache.org followed by helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace. The provider path needs apache-airflow-providers-cncf-kubernetes>=7.4.0 (or pip install 'apache-airflow[cncf.kubernetes]'), and the scheduler needs a non-sqlite backend plus cluster API access.

Values carry executor choice, image tags, git-sync config, scheduler replicas, and resource blocks. Review them like code because they are code: a wrong indentation can drop your resource limits silently. GitOps tools (Argo CD, Flux, Terraform) must set createUserJob/migrateDatabaseJob useHelmHooks=false with applyCustomEnv=false or migrations never run.

Rehearse upgrades in staging with airflow dags test on a canary DAG. Roll back with helm rollback before debugging forward in prod.

 airflowVersion: "3.3.1"
# values.yaml (pinned, production)
executor: "KubernetesExecutor"

images:
  airflow:
    repository: my-registry/airflow
    tag: "3.3.1-python3.12"  # never floating

dags:
  gitSync:
    enabled: true
    repo: "https://github.com/acme/airflow-dags.git"
    branch: prod
    syncInterval: 60

scheduler:
  replicas: 2
  resources:
    requests: {cpu: "500m", memory: "1Gi"}
    limits: {cpu: "2000m", memory: "4Gi"}

Requests and Limits: The OOM Lesson

Requests tell the scheduler what you need; limits tell the kubelet when to kill you. Without requests, pods pack densely until nodes OOM. Without limits, one greedy task eats the node and evicts neighbors. The executor's pod_template_file carries the base image and pod name Airflow requires; everything else layers per task through executor_config (volumes, sidecars, affinity, tolerations) without forking the template.

Set requests to steady-state usage measured over a week, limits to 1.5-2x for bursts. Memory limits kill with OOMKilled; CPU limits throttle. Both beat node-wide eviction cascades. Debug the rendered pod before blaming Airflow: airflow kubernetes generate-dag-yaml dumps exactly what the executor will submit.

Backstop with namespace quotas capping total CPU and pod counts. Quotas turn a runaway backfill into pending pods (visible, bounded) instead of dead nodes (invisible, catastrophic).

KEDA: Scale Workers on Queue Depth

KEDA scales capacity on queue depth instead of CPU. When queued tasks cross thresholds, it adds workers or nodes; when the queue drains, it scales back to near-zero. You pay for burst only during burst.

Point triggers at scheduler queue metrics, not pod CPU. CPU lags task pressure by minutes; queue depth leads it. Thresholds around 50 queued tasks per replica keep waits under 5 minutes without flapping.

Test scale-down as hard as scale-up. Aggressive downscaling that kills warm nodes mid-burst recreates the cold-start storm you installed KEDA to avoid.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: airflow-worker-scale
spec:
  scaleTargetRef:
    name: airflow-worker-pool
  minReplicaCount: 2
  maxReplicaCount: 20
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: airflow_scheduler_queued_tasks
        threshold: "50"  # add capacity per 50 queued tasks

Config via Helm Values vs ConfigMaps

Config arrives via Helm values, ConfigMaps, and secrets, in that precedence order. Values carry structure, ConfigMaps carry files like airflow.cfg snippets, secrets carry connections and keys. Mixing them randomly creates drift nobody can audit. Pod mutation hooks in airflow_local_settings.py add last-mile tweaks (labels, sidecars) without forking the chart.

Git-sync delivers DAGs by polling a branch every 60 seconds. It's flexible across branches but syncs broken commits fleet-wide in one interval. Baked images ship DAGs inside the digest: slower iteration, atomic rollback, zero mid-write parses. Workers read DAG files plus the metadata DB, so either path must reach every scheduler and worker identically.

Gate both with parse checks, and persist logs outside the pod. Without a mounted volume or remote logging, a terminated task pod takes its logs with it. A sync that can't parse shouldn't reach the scheduler, whether it arrives by git or by image.

Cost and Cold-Start Trade-Offs

Cold starts include image pull, pod scheduling, sidecar init, and Airflow task bootstrap. Warm node pools with pre-pulled images cut pulls; slim images cut the rest. Still, 20 seconds is a good day. Fault tolerance leans on a watcher thread: it tails Kubernetes events, marks crashed pods failed, and stores the resourceVersion in the DB so a restarted scheduler resumes the stream instead of rerunning finished tasks. Running tasks report straight to the DB, so scheduler crashes don't fail them.

Cost follows the same curve: per-pod overhead plus node time versus Celery's shared workers. Bursty ML with 30-minute tasks barely notices; hourly 2-minute ETL bleeds money. The official chart can scale Celery workers to zero on queue depth, so the old always-on cost gap narrows when you use it.

Route deliberately with multi-executor setups: steady ETL on Celery queues, spiky training on Kubernetes. One control plane, two cost profiles. Default to the kubernetes queue for pod-bound tasks and keep the rest on Celery workers.