from airflow.operators.bash import BashOperator # Pulling the return value of a TaskFlow task into a Bash script bash_task = BashOperator( task_id="log_demographics", bash_command="echo 'The processed data is: ti.xcom_pull(task_ids=\"process_demographics\") '" ) Use code with caution. 5. Security & Governance: Encrypting and Cleaning XCom Data
Using the execution context or task instance ( ti ) object directly within your operators.
The is not a feature you install; it is a discipline you adopt. By treating XCom as a narrow bridge for references rather than a cargo ship for data, you preserve Airflow’s two greatest strengths: the reliability of the metadata database and the clarity of the task graph. airflow xcom exclusive
: Subclass BaseXCom to override serialize_value and deserialize_value , allowing you to implement custom encryption or specialized compression for sensitive data. 2. TaskFlow API for Clean Scoping XComs — Airflow 3.2.1 Documentation
In older versions of Airflow (or when using classic operators like EmailOperator or BashOperator ), developers must explicitly pull metadata using Jinja templating or by accessing the execution context. from airflow
By default, xcom_pull scopes its search to the current DAG run. However, dynamic tasks or poorly configured task dependencies can inadvertently read historical XCom data. To enforce exclusivity per execution loop, always rely on explicit execution date contexts:
Implement a dedicated maintenance DAG that safely purges historical XCom records using a standard SQL execution or the DbApiHook . The is not a feature you install; it
[core] xcom_backend = include.custom_xcom_backend.S3XComBackend Use code with caution. 4. Advanced XCom Patterns with TaskFlow API
In a multi-tenant environment, you might want to ensure that Task B can pull data from Task A, but Task C (perhaps a notification task) cannot. While Airflow doesn't have native "per-key" permissions, developers implement exclusivity through:
Your specific (AWS, GCP, Azure, or On-Premise)