Airflow XComs: The 10MB Payload That Froze the Database

A task returned a 10MB JSON dict and everything slowed down. The scheduler lagged, the UI timed out, and the metadata DB grew gigabytes in days. The pipeline logic was correct. The channel was wrong.

XComs live in the metadata database, which the scheduler reads constantly. You'll learn why small values fly and large ones freeze the fleet.

We'll cover push and pull patterns, the size trap, and reference passing. Pass pointers, not payloads.

What XComs Are and Where They Live

XCom stands for cross-communication: keyed values tasks exchange through the metadata DB. Each entry carries dag_id, task_id, run_id, key, and a serialized value. Small by design.

Classic tasks push and pull explicitly with xcom_push and xcom_pull. TaskFlow hides the plumbing: returning a value pushes it, and passing task outputs as arguments pulls it. Same store, friendlier syntax.

Treat XCom as a message bus for control data. IDs, URIs, row counts, flags, and tiny dicts belong here. Anything you would hesitate to put in a URL probably does not belong.

Limits come from the metadata DB: Postgres ~1GB, SQLite ~2GB, MySQL ~64KB — and MySQL's 64KB bites fast. Default serialization covers JSON, plus pandas DataFrames (2.6+), Delta and Iceberg tables (2.8+). You'll view rows under Admin > XComs and pull cross-DAG with dag_id plus run_id for TriggerDagRunOperator chains.

Implicit Push and Pull via TaskFlow

Call a @task function and you get an XComArg, not a result. Passing that arg into another task declares both the dependency and the data flow. The scheduler wires the edge; runtime resolves the value.

Returns push automatically under the return_value key. Parameters pull automatically from their upstream args. Explicit xcom_pull remains available for fan-in and dynamic patterns.

This magic stays debuggable. The Graph view shows the edges, and each task instance's XCom tab shows pushed keys. Implicit wiring still leaves visible tracks.

Need fan-out keys? Use @task(do_xcom_push=True, multiple_outputs=True) returning {"key1": ..., "key2": ...} — each key becomes its own XCom you'd pull by key. In templates you'd write SELECT * FROM {{ ti.xcom_pull(task_ids='foo', key='table_name') }}. Note: xcom_pull() with no task_ids only reads the current task in 3.x, and XComs clear on every retry so they can't carry state across tries.

import pendulum
from airflow.sdk import dag, task

@dag(
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    tags=["orders"],
)
def order_refs():
    @task
    def extract() -> str:
        # small reference, not the data itself
        return "s3://forge-staging/orders/2026-09-03.json"

    @task
    def transform(uri: str) -> str:
        print(f"transforming {uri}")
        return "s3://forge-staging/orders/2026-09-03-clean.json"

    @task
    def load(uri: str) -> None:
        print(f"loading {uri}")

    load(transform(extract()))

order_refs()

xcom_pull Patterns: Single, List, by task_id

Pull one value with xcom_pull(task_ids="extract", key="return_value"). Pull fan-in with a list of task_ids. Pull from mapped tasks with map_index. Each form names its source explicitly.

Prefer explicit task_ids over ambient pulls. Unscoped xcom_pull() grabs from the current context's upstream in ways refactors silently rewire. Named sources survive renames with grep-able tracks.

Document keys like API contracts. A producer's return dict is a schema; changing a key breaks consumers exactly like renaming a JSON field.

The Size Trap and DB Bloat

The size trap compounds. One 10MB value times three tasks times daily runs times retained history equals gigabytes in weeks. The xcom table becomes the largest in the metadata DB.

Scheduler queries slow first. Listing runs, rendering Grid, and resolving dependencies all read tables sharing buffers with XCom bloat. Unrelated DAGs stall behind one pipeline's payloads.

Detect with table-size queries and DB growth alerts. The fattest task_ids name the refactor targets. Trend the metric weekly; bloat never announces itself.

For bigger payloads switch backends: set xcom_backend to the object-storage backend (S3/GCS) or a custom BaseXCom subclass with serialize/deserialize plus purge. Verify inside containers with from airflow.sdk.execution_time.xcom import XCom; print(XCom.__name__). CLI helps too: airflow tasks xcom_pull -d <dag> -t <task>.

# find the fattest XComs before they freeze the scheduler
# (Postgres metadata DB)
SELECT dag_id, task_id,
       pg_size_pretty(SUM(pg_column_size(value))) AS total
FROM xcom
WHERE execution_date > NOW() - INTERVAL '7 days'
GROUP BY dag_id, task_id
ORDER BY SUM(pg_column_size(value)) DESC
LIMIT 10;

The Rule: Pass References, Not Data

Reference passing is the whole discipline. Producers write bytes to object storage at versioned paths and return URI strings. Consumers stream by URI. The DB carries sentences, not encyclopedias.

Version the paths with dates or run_ids so reruns and backfills never collide. Add lifecycle rules to expire staging prefixes. Cheap storage with cleanup beats clever in-DB tricks.

Sensors can join the pattern. An XCom-pushed URI becomes the file a downstream sensor waits on, or the partition a quality check counts. Pointers compose; payloads do not.

import pendulum
from airflow.sdk import dag, task
from airflow.providers.amazon.aws.hooks.s3 import S3Hook

@dag(
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    tags=["orders"],
)
def order_uris():
    @task
    def extract() -> str:
        hook = S3Hook(aws_conn_id="forge_storage")
        payload = b'{"orders": 4120}'
        key = "staging/orders/2026-09-03.json"
        hook.load_bytes(payload, key=key, bucket_name="forge-staging")
        return f"s3://forge-staging/{key}"

    @task
    def load(uri: str) -> int:
        print(f"loading from {uri}")
        return 4120

    load(extract())

order_uris()

XCom-Enabled Sensors and When XComs Expire

XCom entries live as long as their runs remain in history. Cleanup purges old runs and their messages together. Nothing expires independently on a timer.

That makes XCom run-scoped by nature. Values needed next quarter belong in tables or buckets with real retention policies. Re-derive or re-read rather than digging through old runs.

Sensors with XCom awareness fit short handoffs: a sensor reads a pushed URI or flag within the same run. Cross-run signaling belongs to assets and datasets, covered later in this course.