Airflow CI/CD: Git-Sync Deployed a Broken DAG Live

The deploy took ninety seconds and broke everything. Git-sync did exactly what it promised: it synced.

The commit it synced was half-written. One DAG file raised at import, and the scheduler stopped parsing all 120 DAGs.

Delivery speed without validation gates is just faster outages. You'll build the gates here.

The Deploy Target: DAGs vs Config vs Providers

Airflow deploys three things: DAG files, configuration, and provider packages. DAGs change daily, config changes weekly, providers change per upgrade. Each needs its own lane with its own gate.

DAGs demand parse-safety above all: no file may raise at import. Config demands validation: lint plus effective-config diffs. Providers demand pinning: identical versions per environment.

Ship them together as one tested unit per release. Version skew between DAGs and providers is a silent prod-only failure.

Git-Sync: Speed With Risk

Git-sync polls a branch and writes files into the live DAG folder every interval. Push to green and prod updates within a minute. Push to broken and prod breaks within a minute. Same mechanism, opposite outcomes. Helm values pin the source: dags.gitSync with repo, branch, rev HEAD, depth 1, wait 60, subPath dags, plus an SSH secret for private repos. VM fleets get the same effect with rsync plus airflow dags reserialize and a Slack ping on success.

The gate makes the difference: sync only commits that passed lint, DagBag, unit, and dag-test. Prod tracks a release branch that fast-forwards solely from green staging builds.

Monitor sync lag and parse errors as deploy metrics. A sync that lands but never parses is a failed deploy wearing success's clothes. Lightweight sidecars (databurst/git-sync with inotify) keep compose-class fleets synced without Kubernetes.

# git-sync sidecar (staging speed, gated commits only)
# values snippet: sync validated branch every 60s
# dags:
#   gitSync: {enabled: true, repo: "https://github.com/acme/airflow-dags.git",
#             branch: staging, syncInterval: 60}

# gate BEFORE sync: must be empty
# airflow dags list-import-errors
# pytest tests/test_dagbag.py -q

Image-Based Deploys: Hermetic and Versioned

Baked images carry DAGs, providers, and config defaults at one digest. Staging tests the exact bytes prod will run; rollback redeploys the prior digest in minutes. Hermetic beats speedy when the scheduler's parsing is at stake.

Build gates live in the Dockerfile: parse checks fail the build on import errors. Label digests with git SHAs so promotion traces to commits.

Keep builds under 10 minutes or teams bypass them. Slim base images and layer caching preserve both safety and velocity.

FROM apache/airflow:3.3.1-python3.12
# hermetic: DAGs + providers baked at one digest
COPY requirements-airflow.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt
COPY dags/ /opt/airflow/dags/
COPY tests/test_dagbag.py /opt/airflow/tests/
RUN airflow dags list-import-errors 2>&1 | tee /tmp/parse.txt && \
    test ! -s /tmp/parse.txt
LABEL org.opencontainers.image.revision="$GIT_SHA"

Branch Promotion and Feature DAGs

Feature DAGs ride feature branches with full CI but no prod sync. Merges to staging trigger dag tests plus staging runs with real connections. Promotion to the release branch requires green staging plus owner approval. A concrete GitHub Actions shape: validate-dags job (checkout, setup-python, pip install apache-airflow plus requirements, validate_dags.py, pytest tests/, flake8 and black), then deploy-staging and deploy-prod jobs gated on it with branch protection requiring CI green before merge.

Keep prod's sync source narrow: one release branch or one digest stream. Wide sources (many branches syncing) multiply unreviewed paths to the scheduler. Never hardcode env specifics in DAGs; promote Variables and Connections through the pipeline per environment instead.

Canary each promotion: one scheduler on the new digest first, watch heartbeats and queue age for 15 minutes, then the fleet. Fifteen patient minutes beat two frantic hours. Roll out the discipline over a month: week 1 version control plus lint, week 2 integrity tests plus staging, week 3 prod deploy with rollback, week 4 monitoring plus canary.

Parse-Safety as a Hard Gate

Parse-safety is the hard gate: airflow dags list-import-errors must print nothing, under empty env, on every candidate commit. Files must import without network, secrets, or host paths. Anything else is a fleet-wide outage waiting for a push. pre-commit plus pylint plus python scripts/validate_dags.py catch syntax before pytest even starts.

Enforce with empty-env CI jobs: env -i pytest tests/test_dagbag.py catches laptop-only imports that full-env CI misses. Lazy-load clients inside tasks so imports stay pure. Common pipeline traps have the same root: dynamic DAG factories untested at generation time, UTC-vs-local timezone drift, DB connections baked into tests, and bloated requirements.txt stretching CI past 10 minutes (fix with layer caching and split dev deps).

Treat gate bypasses as incidents. The urgent hotfix that skips parse checks is statistically the commit that breaks parsing.

Rollback and Canary Runs

Rollback for images is redeploying the prior digest and restarting schedulers. For git-sync it's reverting the branch to the last green commit and waiting one interval. Both need verification: heartbeats fresh, queue age falling, parse errors empty.

Keep three green releases ready and labeled. Purge older ones on a schedule so disk pressure never forces panicked choices.

Run canary checks post-rollback too. A rollback that restores parsing but not connections is half a recovery.