Airflow CI/CD: git-sync Deployed a Broken DAG to Prod

There's a moment in every Airflow platform's life when DAGs stop being pasted into containers and start being deployed. That's the moment the CI/CD decisions get made — and most teams make them in a hurry, because git-sync looks like magic. It syncs your repo to the scheduler, continuously, and DAGs appear with zero deploy steps.

Here's the part the demos skip: git-sync is a file puller, not a release system. It picks up whatever is on the branch at the sync moment — a half-written file mid-commit, a merge that never passed validation, a YAML that broke a provider import. And a broken import in one file can stall scheduling for every DAG on the box.

Deploying Airflow well means separating three things that git-sync happily merges: DAG code, configuration, and provider versions. Each has a different blast radius and a different deploy path. Code goes through review and gates; config goes through rendering and validation; providers go through image builds with pinning.

The fastest deploy is the one you can't roll back.

You want speed you can undo, and that starts with understanding what git-sync actually gives you.

1. The Deploy Target: DAGs vs Config vs Providers

Before choosing a deploy mechanism, separate what's being deployed. DAG code changes constantly and benefits from fast iteration. Configuration changes rarely but breaks everything when wrong. Provider versions change on a release cadence and silently alter operator behavior.

These three have different blast radii. A bad DAG breaks one pipeline — unless it fails at import, in which case it can degrade the whole scheduler. A bad config breaks every component that reads it. A provider mismatch breaks every DAG that uses that provider.

Different blast radii demand different deploy paths. DAGs: fast path with gates. Config: rendered, validated, diffed. Providers: pinned in an image, promoted only deliberately.

The first CI/CD decision isn't which tool — it's which of these three you're shipping. Most git-sync incidents, including ours, are failures of this separation: everything treated as DAG code.

2. git-sync: Speed With Risk

git-sync is a sidecar container that polls a git repo and copies the checked-out files into a shared volume — the scheduler's dags folder. DAGs land seconds after merge. No builds, no registry, no deploy step. It's the fastest path DAGs have ever had, and it's seductive.

The risk is the mechanism itself: it pulls whatever the branch holds at the sync moment. That includes WIP commits, half-written files mid-push, and merges that never ran a single test. The repo is not a release channel — it's a working tree with history.

The mitigation is the gate, and it must sit between the branch and the sync: branch protection, parse validation, and promotion to a protected branch that the sync follows. The sync follows main; main only ever receives validated commits.

git-sync also needs a rollback story. Pin the sync to a commit (git-sync's SYNC_REVISION), not just to a branch — then rollback is one restart with an older pin, not an emergency revert race against the next sync tick.

Two mechanics are worth knowing. First, the checkout is atomic: git-sync clones into a fresh folder and swaps a symlink when the checkout finishes, so the dags folder always presents one consistent commit — never a byte-level mix of two. Second, scope: git-sync follows exactly one repository, the sidecar runs on every dag-processor, worker, and triggerer pod when persistence is off, and with KubernetesExecutor it becomes an init container on worker pods. Multi-repo teams use git submodules in an umbrella repo, and the default sync interval is 60 seconds (gitSync.period in the Helm chart).

# git-sync sidecar for the scheduler — pinned, not floating
- name: git-sync
  image: registry.k8s.io/git-sync/git-sync:v4.2.4
  args:
    - --repo=https://github.com/acme/airflow-dags
    - --branch=production        # promoted branch, never feature branches
    - --rev=$(GIT_SYNC_REV)      # pin: rollback = change this one value
    - --root=/dags
    - --wait=30
  env:
    - name: GIT_SYNC_REV
      value: "v2026.07.15.1"     # exact commit or tag

# Rollback: restart the scheduler with GIT_SYNC_REV=v2026.07.14.2
# — no revert commit, no race against the next poll.

3. Image-Based Deploys: Hermetic, Versioned

The image-based path treats DAGs like application code: they're pip-installed into the Airflow image at build time, the image is tagged with a version, and deploying means rolling pods to that tag. The image is a snapshot — everything in it was built together and validated together.

Hermetic is the point. The scheduler and workers run the exact DAG bytes, config, and provider versions that passed the build. No sync window, no branch ambiguity, no mid-write files. A build that fails validation never becomes an image; an image that's bad is never rolled out.

The cost is iteration speed. Each deploy is a build, a registry push, and a pod roll — minutes, not seconds. For teams where DAG changes are release-shaped (weekly, or gated by compliance), that's the right price. For teams doing ten deploys a day, the build queue becomes the bottleneck.

Versioning discipline is non-negotiable: immutable tags (image:2026.07.15-1), never :latest, and a promotion path where staging and prod reference different, recorded tags. Rollback is then one command — roll pods to the previous tag.

# DAGs pip-installed into the image — hermetic deploys
FROM apache/airflow:3.3.0-python3.12

USER root
# Providers pinned at build time — no drift between scheduler and workers
COPY requirements-airflow.txt /requirements-airflow.txt
RUN pip install --no-cache-dir -r /requirements-airflow.txt

# DAGs become a real package — tested, then installed
COPY dags/ /opt/airflow/dags/
COPY airflow_dags_pkg/ /opt/airflow/airflow_dags_pkg/
RUN pip install --no-cache-dir /opt/airflow/airflow_dags_pkg

USER airflow

# Build:   docker build -t acme/airflow:2026.07.15-1 .
# Promote: roll scheduler+workers to acme/airflow:2026.07.15-1
# Rollback: roll to acme/airflow:2026.07.14-3  (previous immutable tag)

4. Branch Promotion and Feature DAGs

Branch promotion is the discipline that makes either deploy path safe: feature branches merge to staging, staging merges to production, and production is the only branch the sync (or image tag) may follow. Every hop runs the gate again, because the merge that's safe on staging may not be on prod.

Feature DAGs — DAGs developed on a branch — need a special rule: they must be parse-safe even when incomplete. The trick is that feature work lives in the repo but never lands on the synced branch until it's finished, and while it's on a feature branch, nothing is expected to be runnable. The production branch is sacred: nothing merges to it that hasn't passed the gate.

The promotion hops double as your canary. Staging merges run the full suite; production merges re-run the parse gate and dags test on the exact production image. If a DAG breaks at the staging hop, you've caught it a full deploy early.

Promotion also gives you auditability: every prod DAG version traces to a merge commit, and every rollback traces to a tag.

One documented pattern for multi-cluster setups: point staging and prod at the same synced branch and control what each deployment loads with AirflowClusterPolicySkipDag (added in 2.7) — a cluster policy raises to exclude specific DAGs from that cluster's DagBag, instead of maintaining parallel branches that drift. That's what "release-shaped" means for Airflow.

5. Parse-Safety as a Hard Gate

Parse-safety is the one property every DAG file must have, unconditionally: importing the file must never fail, never block, never touch the network or the database. The scheduler imports every file on every loop; a file that fails at import degrades or stops the whole loop. That's the exact mechanism that took our scheduler down.

The gate enforces it in CI: load every DAG through a DagBag, assert zero import_errors, and enforce the no-side-effects rule at review time (top-level code pure, imports lazy, connections resolved at runtime, not import time).

Parse-safety also has a runtime form: files should be structured so a partial file is still importable — a syntax error in a WIP branch is fine on a feature branch, but the merge gate must be brutal about it.

If a DAG can't fail at import, the worst case shrinks from fleet-wide stall to one failed run. That single property is worth more than every other deploy optimization combined.

# tests/test_parse_safety.py — the hard gate, every merge
import pytest
from airflow.models import DagBag


def test_all_dags_import_without_errors():
    dagbag = DagBag(dag_folder="dags/", include_examples=False)
    assert not dagbag.import_errors, f"{dagbag.import_errors}"


def test_parse_side_effect_free():
    # Importing a DAG must not touch the network or a database.
    # The classic violation: a top-level Hook or connection lookup.
    import ast

    with open("dags/market_etl.py") as fh:
        tree = ast.parse(fh.read())

    top_level = [n for n in ast.walk(tree) if isinstance(n, ast.Call)]
    banned = ["SnowflakeHook", "PostgresHook", "requests.get", "get_connection"]
    assert not any(
        getattr(c.func, "attr", "") in banned or getattr(c.func, "id", "") in banned
        for c in top_level
    )

6. Rollback and Canary Runs

Every deploy path needs an undo that's faster than the outage it's undoing. For image deploys, rollback is rolling pods to the previous immutable tag — minutes, deterministic. For git-sync, rollback is restarting the sync at a pinned commit — seconds, but only if you pin. Both fail if the rollback path was never rehearsed.

Canary runs are the second safety net: after a deploy, trigger the highest-value DAG once and watch its state before letting the whole schedule run on the new code. A canary that fails stops the rollout with the blast radius of one DAG, not one fleet.

Combine both in the runbook: deploy → canary run → observe one full cycle → release the schedule. The canary is the difference between discovering a broken deploy from a DAG run and discovering it from a pager alert.

Rehearse rollback the way you rehearse failover — in staging, on a schedule, with a stopwatch. The first time you roll back should not be during the incident.

# After every deploy — canary first
airflow dags trigger market_etl --run-id canary-2026-07-15
sleep 60
airflow dags list-runs market_etl --state failed
# zero failed = safe to release the schedule

# git-sync rollback: one env value, one restart
GIT_SYNC_REV=v2026.07.14.2   # pinned back to the last good commit
kubectl rollout restart deployment/airflow-scheduler

# image rollback: one tag
kubectl set image deployment/airflow-scheduler \
  scheduler=acme/airflow:2026.07.14-3
kubectl rollout status deployment/airflow-scheduler

7. Incident Deep Dive: Failure Flow → Fix

Replay our afternoon. A developer pushed a WIP commit to main — the branch git-sync followed — with a DAG file saved mid-edit: an unfinished import statement and a broken reference. No gate stood between the branch and the sync; the push was the deploy.

At the next poll interval, the sidecar copied the half-written file into the scheduler's dags folder. The scheduler's parse loop hit it, raised ImportError, and kept hitting it every loop. With one file failing at import, the scheduler's parsing degraded, scheduling slowed, and then stopped — every DAG on the box stalled, including the ones with nothing to do with the broken file.

The failure flow is the lesson: no validation between push and sync, no pinned revision, no canary — one WIP commit was a production deploy with no undo.

The fix rebuilt the path in four layers. First, branch protection: the sync follows production, and production only receives promoted, gated merges. Second, the parse-safety gate: DagBag with zero import_errors, plus a side-effect check, running at every promotion hop. Third, pinning: the sidecar syncs to an exact revision, so rollback is a restart with the previous pin. Fourth, canaries: every deploy triggers market_etl once and watches it land before the schedule opens.

git-sync didn't betray us — it did exactly what it does: it pulled bytes. We had asked it to be a release manager. Now we know the difference.