Airflow XComs: The 10MB Payload That Froze the Scheduler DB

XComs look like the easiest way to move data between tasks. They're also the most common way to slowly kill a production scheduler. Every value a task pushes lands in the metadata DB, and the metadata DB is the same database the scheduler and the web UI hammer on every heartbeat. Bloat it and everything slows down at once.

We had a task that built a 10MB JSON dict of order summaries and returned it straight to XCom. Harmless-looking — one return statement. Inside the scheduler it was a truck dumped into a filing cabinet, and the cabinet started jamming. The UI crawled, task instance queries took seconds, and the DB grew every single run.

Here's what XComs are, how push and pull really work, and the one rule that keeps your scheduler fast: pass references, not data. It's a small rule with a 10MB lesson behind it.

1. What XComs Are (and Where They Live)

XCom is short for cross-communication. A task pushes a key-value pair into a table in the metadata DB, and any other task in any DAG can pull it back by dag_id, task_id, and key. That's the whole mechanism — a shared key-value table.

The "where they live" part is the part everyone skips: the xcom table lives in the same PostgreSQL (or SQLite) database that stores every DAG run, task instance, and scheduler state. The scheduler reads that database continuously, and the web UI reads it on every page load.

So XCom is not a data pipeline. It's a control channel that happens to be backed by your most load-sensitive table. Treat it accordingly, and the whole failure class disappears.

2. Implicit Push/Pull via TaskFlow

TaskFlow made XCom invisible. A @task function's return value is automatically pushed to XCom under the key return_value, and passing that result as an argument to another @task automatically pulls it. Zero explicit code — and zero awareness of the DB write happening under the hood.

That invisibility is the trap. Teams write pipelines that return pandas DataFrames, lists of dicts, and serialized models, never once looking at the XCom tab. The scheduler is writing every one of those to the metadata DB, and pickling big objects per run.

Implicit is convenient and dangerous. The moment a return value gets big, the convenience becomes a cost center. Know what your tasks return, and keep returns small by design.

from airflow.decorators import dag, task
from airflow.utils.dates import days_ago

@dag(
    schedule="@daily",
    start_date=days_ago(2),
    catchup=False,
    default_args={"retries": 2},
)
def nightly_orders():
    @task
    def count_orders():
        # small value: pushed to XCom as return_value
        return 4_320

    @task
    def send_report(order_count: int):
        # pulled from XCom automatically
        print(f"Orders processed: {order_count}")

    send_report(count_orders())

nightly_orders()

3. xcom_pull Patterns: Single, List, by task_id

Classic operators pull explicitly with ti.xcom_pull. The three patterns you'll actually use: pull one task's return_value, pull a specific key, and pull multiple task_ids to get a list of values.

Each pattern has an exactness trap. A typo in task_id or key returns None — no error, no warning, just a quiet None that flows downstream. This is the XCom failure mode that takes longest to find, because the DAG stays green while the data rots.

The fix is the same as everywhere else in this series: explicit keys, typed returns, and a unit test that proves the pushed value is what the pulling task expects. XCom is a contract between tasks; contracts deserve tests. Two more patterns are worth keeping handy: return a dict with multiple_outputs=True (and do_xcom_push=True) and each key is pushed as its own XCom row — pullable by key, while a keyless xcom_pull still defaults to return_value. Also note the Airflow 3 behavior change: xcom_pull() without task_ids now pulls only from the current task, where Airflow 2 searched all tasks and returned the most recent value — always pass task_ids explicitly when pulling from another task.

from airflow.operators.python import PythonOperator


def collect_counts(**context):
    ti = context["ti"]

    # pull the default return_value from one task
    total = ti.xcom_pull(task_ids="count_orders")

    # pull a specific key from a task
    by_region = ti.xcom_pull(task_ids="group_orders", key="by_region")

    # pull the return_value from several tasks into a list
    all_counts = ti.xcom_pull(
        task_ids=["count_orders", "count_refunds", "count_failures"]
    )
    return sum(all_counts)

4. The Size Trap and DB Bloat (Incident Deep Dive)

The incident task returned a 10MB JSON dict of order summaries. In isolation it looked fine — one task, one run. But XCom persists, so that 10MB sat in the xcom table, and every scheduler loop, every UI grid refresh, and every downstream pull read the table. Ten megabytes of tax, every read, every run, compounding.

Within days the DB had grown, queries that took milliseconds took seconds, and the scheduler fell behind. The failure wasn't a crash. It was death by slow read — the hardest outage to see coming, because nothing is red.

Airflow does offer custom XCom backends — including an official object-storage backend that keeps values out of the database entirely — but those are painkillers, not cures. The cure is size discipline at the source: return IDs, counts, and URIs. The 10MB dict belonged in S3, not the scheduler's spine.

import json
from airflow.decorators import task

MAX_XCOM_BYTES = 4_096  # stay tiny: XCom lives in the metadata DB

@task

def summarize_orders(order_summaries: dict) -> str:
    payload = json.dumps(order_summaries).encode()
    if len(payload) > MAX_XCOM_BYTES:
        raise ValueError(
            f"XCom payload too large ({len(payload)} bytes) — "
            f"write to object storage and pass the URI instead"
        )
    return f"{len(order_summaries)} regions summarized"

5. The Rule: Pass References, Not Data

The rule that fixes this class of incident for good: XCom carries references, never payloads. Write the heavy artifact — the DataFrame, the JSON export, the model — to object storage, and push the URI.

Why object storage and not the DB? Three reasons: S3/GCS is built for large blobs and infinite retention; the metadata DB is built for rows of state and slow reads. Object storage costs pennies per gigabyte; the scheduler's DB costs you every query. And a URI is debuggable — you can go look at the object, re-download it, and inspect it.

This pattern is idiomatic Airflow 3: write to object storage, pass the URI, read it in the next task. Same shape as the incident's fix, minus the month of pain.

import boto3
from airflow.decorators import task

S3_BUCKET = "market-etl-artifacts"

@task
def export_order_summaries(order_summaries: dict) -> str:
    s3 = boto3.client("s3")
    key = "orders/summaries_2026_08_01.json"
    s3.put_object(Bucket=S3_BUCKET, Key=key, Body=json.dumps(order_summaries))
    return f"s3://{S3_BUCKET}/{key}"   # the URI, not the payload

@task
def load_summaries(summary_uri: str):
    s3 = boto3.client("s3")
    bucket, key = summary_uri.replace("s3://", "").split("/", 1)
    body = s3.get_object(Bucket=bucket, Key=key)["Body"].read()
    print(f"Loaded {len(body)} bytes from {summary_uri}")

6. XCom-Enabled Sensors and When XComs Expire

Sensors can pass data too. A sensor task can push what it found — a file path, a query result, a row count — into XCom for the next task to consume. It's the same mechanism with a sensor-shaped entrance.

Then there's the lifecycle question: when do XComs go away? They're cleared with their task instances and DAG runs — clear a run and its XComs go with it. Old runs get cleaned up by the data retention policy on dag runs, and XComs without cleanup accumulate: that's part of the bloat math above. One doc-level rule most teams learn the hard way: if a task's first attempt fails, its XComs are cleared on every retry so the run stays idempotent — XComs cannot be used to persist state across task retries or across sensor pokes.

Know what your XComs are before you need to clean them. A quick query on the xcom table by size and age tells you who's using the channel responsibly and who's treating your scheduler's filing cabinet as a data warehouse.