5 Complex task dependencies
This chapter covers:
● Examining how to differentiate the order of task dependencies in an Airflow DAG.
● Explaining how to use trigger rules to implement joins at specific points in an Airflow DAG.
● Showing how to make conditional tasks in an Airflow DAG, which can be skipped under certain conditions.
● Giving a basic idea of how trigger rules function in Airflow and how this affects the execution of your tasks.
In previous chapters, we’ve seen how to build a basic DAG and define simple dependencies between tasks. In this chapter, we will further explore exactly how task dependencies are defined in Airflow and how these capabilities can be used to implement more complex patterns including conditional tasks, branches and joins. Towards the end of the chapter we’ll also dive into XComs, which allow passing data between different tasks in a DAG run, and discuss the merits and drawbacks of using this type of approach.