5 Defining dependencies between tasks
This chapter covers
- Defining task dependencies in an Airflow DAG
- Implementing joins using trigger rules
- Making tasks conditional on certain conditions
- How trigger rules affect the execution of your tasks
- Using XComs to share state between tasks
- Simplifying DAGs with the Airflow Taskflow API
In previous chapters, we saw how to build a basic DAG and define simple dependencies between tasks. In this chapter, we will further explore exactly how task dependencies are defined in Airflow and how these capabilities can be used to implement more complex patterns, including conditional tasks, branches, and joins. Toward the end of the chapter, we’ll also dive into XComs (which allow passing data between different tasks in a DAG run), and discuss the merits and drawbacks of using this type of approach. We’ll also show how Airflow Taskflow API can help simplify DAGs.
5.1 Basic dependencies
Before going into more complex task dependency patterns such as branching and conditional tasks, let’s first take a moment to examine the different patterns of task dependencies that we’ve encountered in the previous chapters. This includes both linear chains of tasks (tasks that are executed one after another), and fan-out/fan-in patterns (which involve one task linking to multiple downstream tasks, or vice versa). To make sure we’re all on the same page, we’ll briefly go into the implications of these patterns in the next few sections.