Airflow: Xcom Exclusive

If exclusive XCom access is critical for correctness, consider:

@task def produce_id() -> str: return "data_lake/2024-01-01/partition.parquet"

An XCom entry is uniquely identified by: airflow xcom exclusive

Many Airflow operators automatically push their return value to XCom using the default key return_value when do_xcom_push=True (which is the default setting). @task functions also automatically push their return values to XCom.

: Because XComs live in your metadata database (like Postgres), they are typically limited to 1 GB . If exclusive XCom access is critical for correctness,

Instead of relying on implicit XCom generation, explicitly define which task pushes data and which task pulls it. This creates a clear, traceable data lineage in your code. Using the TaskFlow API for Exclusive Mapping

To configure a custom backend, you must create a custom class inherited from BaseXCom and implement the serialize and deserialize methods. Instead of relying on implicit XCom generation, explicitly

def task_a(**context): context['ti'].xcom_push(key=f"result_context['ti'].task_id", value=100)

Airflow does not provide a built-in “consume-once” XCom primitive in older versions; there are two main approaches:

: Rely on XCom only for small, idempotent, non-critical data. For exclusive workflows, redesign your DAG or bring your own locking mechanism.

To activate your new backend, expose it to Airflow via an environment variable or edit your airflow.cfg .