What Are XComs?
XComs (short for cross-communications) let tasks in a DAG exchange small pieces of data during a DAG run. Each XCom is stored as a row in Airflow's metadata database, keyed by task_id, dag_id, run_id, and a key string, and is meant for small serializable values such as a file path, a row count, or a JSON-serializable dict — not bulk datasets or dataframes.
Cricket analogy: It's like the twelfth man passing a specific note ('bowl short to the tailender') from the dressing room to the captain between overs — a tiny targeted message, not the entire scouting dossier.
Pushing and Pulling XCom Values
Tasks push XComs either explicitly via ti.xcom_push(key='...', value=...) inside a task's execution context, or implicitly when a callable's return value is automatically stored under the default key return_value. Downstream tasks retrieve values with ti.xcom_pull(task_ids='upstream_task_id', key='return_value'); passing a list of task_ids to task_ids returns a list of matching values in the same order, which is useful when fanning in results from parallel tasks.
Cricket analogy: It resembles a fielding coach explicitly radioing 'field is set for a pull shot' (an explicit push) versus the scoreboard automatically updating after every ball (an implicit push).
from airflow.decorators import dag, task
from datetime import datetime
@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False)
def xcom_demo():
@task
def extract() -> dict:
# Return value is auto-pushed to XCom under key 'return_value'
return {"row_count": 4213, "source": "orders_api"}
@task
def load(stats: dict):
print(f"Loading {stats['row_count']} rows from {stats['source']}")
# TaskFlow wires the XCom push/pull automatically
load(extract())
# Classic operator equivalent using explicit pull:
# def _load(**context):
# ti = context["ti"]
# stats = ti.xcom_pull(task_ids="extract", key="return_value")
# print(stats["row_count"])
xcom_demo()XComs with the TaskFlow API
The TaskFlow API (the @task decorator) removes most manual XCom bookkeeping: when one @task-decorated function's return value is passed as an argument to another, Airflow automatically wires an xcom_push on the producer and an xcom_pull on the consumer, and it correctly serializes dicts, lists, and other JSON-safe structures. Multiple return values can be unpacked using multiple_outputs=True, which stores each dictionary key as a separate XCom entry rather than one blob, letting downstream tasks selectively pull only what they need.
Cricket analogy: It's like a well-drilled fielding unit where the wicketkeeper's signal to the bowler happens automatically through practiced routine, no explicit shouting needed, unlike a rookie side that must call everything out loud.
Custom XCom Backends and Size Limits
By default, XComs are pickled or JSON-serialized and stored directly in the metadata database, which is fine for kilobyte-sized payloads but becomes a bottleneck or outright failure point for anything large, since databases like Postgres and MySQL have practical row-size and performance limits for this kind of usage. For larger payloads, Airflow supports a custom XCom backend (configured via core.xcom_backend) that intercepts serialize_value/deserialize_value calls and instead writes the actual data to object storage such as S3 or GCS, storing only a lightweight reference (like a URI) in the metadata database.
Cricket analogy: It's like a stadium's scoreboard system storing just the score summary in its display memory while the full ball-by-ball video feed is archived on a separate broadcast server.
The default database-backed XCom size limit depends on your metadata DB: Postgres can technically store large values but performance degrades quickly past a few KB-MB, while MySQL's default TEXT/BLOB columns cap out around 64KB unless altered. As a rule of thumb, treat XComs as suitable for values under a few hundred KB and use a custom backend or external storage for anything larger.
Never push a full pandas DataFrame, a large file's raw bytes, or an entire API response payload through XCom. This bloats the metadata database, slows down the scheduler and webserver (which both query that table), and can cause task failures. Instead, write the data to S3/GCS/a shared filesystem and pass only the path or URI through XCom.
- XComs let tasks pass small, serializable pieces of data between each other within a DAG run.
- A task's return value is automatically pushed to XCom under the key 'return_value'; ti.xcom_push() allows explicit custom keys.
- ti.xcom_pull(task_ids=..., key=...) retrieves values; passing a list of task_ids returns a list of results.
- The TaskFlow API (@task) auto-wires XCom push/pull when you pass one task's return value into another.
- multiple_outputs=True splits a dict return value into separate XCom entries, one per key.
- XComs are stored in the metadata database by default and are not meant for large data.
- Use a custom XCom backend (e.g., S3) to store large payloads externally and keep only a reference in the DB.
Practice what you learned
1. What is the default key under which a task's return value is stored as an XCom?
2. Where are XComs stored by default in Airflow?
3. In the TaskFlow API, how do you split a dictionary return value into separate XCom entries per key?
4. What is the recommended pattern for passing a large dataset between tasks?
Was this page helpful?
You May Also Like
Variables and Templating
Learn how Airflow Variables provide runtime configuration and how Jinja templating lets task parameters be computed dynamically at execution time.
Hooks and Connections
Understand how Airflow Connections store credentials for external systems and how Hooks provide a Python interface to interact with them.
Branching with BranchPythonOperator
Learn how to build conditional paths in a DAG using BranchPythonOperator, and how Airflow's trigger rules interact with skipped tasks.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics