Airflow on Kubernetes: 400 Pods, 3 Nodes, 1 Nightmare

KubernetesExecutor is Airflow's most isolated execution model: every task instance gets its own pod. No shared filesystems, no neighbor's cron job, no cross-worker state. It's a superpower and a trap.

The trap is resources. Kubernetes schedules pods, but it doesn't size them. A task pod with no requests and no limits is an unbounded bet on a finite cluster, and the cluster always collects.

The official Helm chart gets you a working stack in one command. The problem is everything after the command: resources, scaling, DAG delivery, and the version you pinned to.

Treat the cluster like a finite budget. Every pod is a line item.

1. KubernetesExecutor: A Pod Per Task

KubernetesExecutor runs every task instance in its own pod. The scheduler creates the pod, the pod runs the operator code, and the pod disappears. Isolation per task is the whole design. You enable it in airflow.cfg with executor = KubernetesExecutor, set parallelism, and let the cluster do the rest.

That isolation buys you a lot: a memory-hungry task can't kill its neighbors, resource-hungry jobs get dedicated requests, and the cluster handles retries by creating fresh pods.

It costs you startup time. Every task pays a pod scheduling and image-pull warm-up, seconds per task that Celery workers don't pay. High-frequency short tasks bleed throughput here.

The trade-off is clean: isolation and elastic capacity against cold starts and cluster complexity. Choose it when tasks are heavy, varied, or untrustworthy.

The pod is created when the task is queued and terminates when the task completes. You don't have to pick one executor either: CeleryKubernetesExecutor runs both on one cluster — tasks on the kubernetes queue get their own pod, everything else goes to Celery workers — and KubernetesPodOperator gives you a per-task pod under any executor.

2. Helm Install of the Official Chart

The official Apache Airflow Helm chart deploys the entire stack: scheduler, webserver, workers, postgres, and redis, all as one release. One helm install, one values.yaml, and a production-grade cluster appears.

The chart's real value is templating: workers, resource settings, git-sync, and KEDA support are all values. You configure the platform in YAML instead of hand-rolling manifests.

Pin the version. Unpinned charts drift: a minor bump silently changed our scheduler pod template and the fleet behaved differently overnight. Chart upgrades deserve the same review as code changes.

Install once, pin forever, diff every upgrade.

helm repo add apache-airflow https://airflow.apache.org
helm repo update

helm install airflow apache-airflow/airflow \
  --namespace airflow --create-namespace \
  --version 1.15.0 \
  --values values.yaml

# Verify the release and the running pods
helm list -n airflow
kubectl get pods -n airflow

3. Requests and Limits: The OOM and Eviction Lesson

Requests and limits are the two numbers every pod needs. requests are a promise: the scheduler only places the pod where that capacity is free. limits are a ceiling: the pod never exceeds them, even under pressure.

No requests means the scheduler sees zero cost and packs pods anywhere. No limits means a pod can consume a whole node and get OOM-killed when the kubelet pushes back.

Evictions are the painful half of the story. When a node runs out of memory, kubelet evicts pods, often the very ones doing real work. A 400-pod backfill on 3 nodes is the perfect eviction machine.

Set both, per task. Requests keep scheduling honest; limits keep one pod from eating the node.

For per-task settings the lighter lever than pod_override is executor_config with the KubernetesExecutor keys request_cpu, limit_cpu, request_memory, limit_memory. Two field notes: the scheduler always overwrites metadata.name and containers[0].args on any V1Pod you supply, and tight CPU or memory limits make pod startup slower — size limits from measured usage, not guesses.

from kubernetes.client import models as k8s
from airflow.operators.python import PythonOperator

pod = k8s.V1Pod(
    spec=k8s.V1PodSpec(
        containers=[
            k8s.V1Container(
                name="base",
                resources=k8s.V1ResourceRequirements(
                    requests={"cpu": "250m", "memory": "512Mi"},
                    limits={"cpu": "2", "memory": "4Gi"},
                ),
            )
        ],
    )
)


def flatten_market_data():
    pass


flatten = PythonOperator(
    task_id="flatten_market_data",
    python_callable=flatten_market_data,
    pod_override=pod,
    dag=dag,
)

4. KEDA: Scale Workers on Queue Depth

Static scheduler replicas are a gamble: too few, and a backfill starves; too many, and you pay for idle pods. KEDA closes the loop by scaling on the thing that matters: queue depth.

A ScaledObject watches the Celery queue and adjusts the scheduler or worker Deployment between minReplicaCount and maxReplicaCount. Deep queue, more replicas. Empty queue, fewer. No human, no guesswork.

This is queue-based autoscaling, which is different from HPA's CPU targeting. HPA reacts to resource pressure after it happens; KEDA reacts to work actually waiting.

Set the trigger threshold to your queue budget. A threshold of 10 means ten queued tasks add a replica, and the fleet follows the backlog.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: airflow-scheduler
  namespace: airflow
spec:
  scaleTargetRef:
    kind: Deployment
    name: airflow-scheduler
  minReplicaCount: 1
  maxReplicaCount: 4
  triggers:
    - type: redis
      metadata:
        address: redis.airflow.svc:6379
        listName: airflow.executor.celery
        listLength: "10"

5. Config via Helm Values vs ConfigMaps

Two config surfaces exist on Kubernetes: Helm values for chart-managed settings, and ConfigMaps for Airflow's own configuration files.

Helm values own the deployment: executor, resource limits, git-sync, KEDA, replica counts. Change them, helm upgrade, and the chart reconciles the fleet. A values.yaml entry like executor: "KubernetesExecutor" or scheduler.replicas: 2 reshapes the release.

ConfigMaps own Airflow's config: airflow.cfg overrides and environment variables. Mount them into pods, and the running Airflow reads them without a chart cycle. Airflow-specific settings travel as AIRFLOW__SECTION__KEY variables.

The rule is separation of concerns: deployment shape in values, application config in ConfigMaps, secrets in Secrets, never in either. Mixing them turns every upgrade into archaeology.

6. Git-Sync DAGs on Kubernetes

Pods are ephemeral; DAGs must not be. Git-sync solves delivery: a sidecar container clones your DAG repository into the shared DAG folder on an interval.

The scheduler and worker pods each run a git-sync sidecar alongside the main container. Merge to the configured branch, and DAGs propagate without rebuilding images or copying files.

The price is discipline: git-sync ships whatever is on the branch, including half-written files. Parse-safety gates in CI are the only thing standing between a bad commit and a dead scheduler.

Add validation to CI before merge: dag loading tests catch the broken imports that otherwise take the whole fleet down at once.

dags:
  gitSync:
    enabled: true
    repo: https://github.com/acme/data-platform-dags.git
    branch: main
    rev: HEAD
    subPath: dags
    syncIntervalSec: 60

  persistence:
    enabled: true

7. Cost and Cold-Start Trade-offs

Pod-per-task pays a cold-start tax every run: image pull, scheduling, container start. A fleet of short tasks pays that tax constantly, and the cluster bill grows with every backfill.

Cost follows the same curve. Each pod consumes CPU and memory while it runs, and idle scheduler replicas burn money for nothing. KEDA shrinks the idle fleet, but pod overhead remains.

The honest pattern: KubernetesExecutor for heavy, isolated, or bursty tasks; CeleryExecutor for the steady stream of small ones. Both can live under one scheduler.

Match the executor to the task, and the bill follows the work.

Pods also survive deploys: if you roll out while a task runs, KubernetesExecutor lets the pod finish, while Celery only waits out the configured grace period before terminating in-flight tasks.