liveBook · Manning

Task group

As explained in Data Pipelines with Apache Airflow, Second Edition

An Airflow task group is a grouping abstraction that organizes related tasks and presents them as a condensed unit in the workflow interface.

A task group organizes related Airflow tasks into a smaller visual unit, making complex DAGs easier to inspect without hiding access to their individual tasks. Use the TaskGroup context manager to group tasks generated for each dataset, such as sales or customers. section 12.1.6

Group dataset-processing tasks

for dataset in ["sales", "customers"]: with TaskGroup(dataset, tooltip=f"Tasks for processing {dataset}"): generate_tasks( dataset_name=dataset, raw_dir="/data/raw", processed_dir="/data/processed", output_dir="/data/output", preprocess_script=f"preprocess_{dataset}.py", dag=dag, ) The web interface initially shows each group as one condensed node. Expand a group to inspect its tasks; nested task groups are also supported. section 12.1.6

Listing 12.29 Using TaskGroups to group tasks visually
from airflow.sdk import DAG, TaskGroup
...
for dataset in ["sales", "customers"]:
    with TaskGroup(dataset, tooltip=f"Tasks for processing {dataset}"):
        generate_tasks(
            dataset_name=dataset,
            raw_dir="/data/raw",
            processed_dir="/data/processed",
            output_dir="/data/output",
            preprocess_script=f"preprocess_{dataset}.py",
            dag=dag,
    )

Browse the complete TaskGroup listing showing dataset-specific grouping, tooltips, and generated task parameters. section 12.1.6

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Explains the context-manager implementation and expandable interface behavior, but does not detail task-group execution internals.
Examples
Uses sales, customers, weather, and movie-review processing examples, with Python listings and Airflow interface figures.
What it assumes
Assumes familiarity with Airflow DAGs and tasks while demonstrating grouping from generated dataset-processing tasks.
What you can do after
Apply it via TaskGroup or @task_group to reduce DAG visual noise while retaining expandable access to individual tasks.
How it connects
Connects task groups with DAGs, factory functions, nested task groups, and Dynamic Task Mapping.
Cover of Data Pipelines with Apache Airflow, Second Edition
liveBookmanning.com
Read Data Pipelines with Apache Airflow, Second Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.