liveBook · Manning

Expand()

As explained in Data Pipelines with Apache Airflow, Second Edition

expand() is an Airflow task-mapping method that specifies the iterable values for which separate mapped task instances are created.

When a workflow discovers an unknown number of movie reviews, expand() lets Airflow create one mapped task instance for each review instead of processing them in a loop. section 12.1.8

Map a task over fetched reviews

Keep shared operator arguments in partial(), then supply the mapped values through expand(): PythonOperator.partial(task_id="print_rating", python_callable=_print_rating).expand(op_args=fetch_ratings.output). The fetch task runs first; Airflow creates and independently processes one print task for each returned review.

Each DAG run can produce a different number of instances. Airflow exposes mapped tasks in the grid and graph views with [] notation and individual instances.

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Explains syntax, dependencies, TaskFlow usage, TaskGroups, and UI visibility without detailing scheduler internals.
Examples
Demonstrates adding integers and printing movie reviews fetched from an API.
What you can do after
Build dynamically mapped Airflow tasks that process runtime-generated values independently.
How it connects
Connects Dynamic Task Mapping with partial(), XCom, PythonOperator, TaskFlow API, and TaskGroups.
Cover of Data Pipelines with Apache Airflow, Second Edition
liveBookmanning.com
Read Data Pipelines with Apache Airflow, Second Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.