2 Anatomy of an Airflow DAG

In the previous chapter, we learned why working with data and the many tools in the data landscape is not an easy task. In this chapter we get started with Airflow and check out an example workflow that uses basic building blocks found in many workflows. It helps to have some Python experience when starting with Airflow; since workflows are defined in Python code, the gap to learning the basics of Airflow is not that big. Getting a workflow up and running with Airflow is often not a hard task; the number of concepts to learn for a newcomer is low. The more complicated part is knowing when to, and when not to make certain choices; something that typically comes with hands-on experience.

2.1   Tracking rocket launches

2.1.1   Launch Library

2.2   Writing your first Airflow DAG

2.2.1   Tasks vs operators

2.2.2   Running arbitrary Python code

2.3   Running a DAG in Airflow

2.4   Running at regular intervals

2.5   Handling failing tasks

2.6   Summary