2 Anatomy of an Airflow DAG

In the previous chapter, we learned why working with data and the many tools in the data landscape is not easy. In this chapter, we get started with Airflow and check out an example workflow that uses basic building blocks found in many workflows.

It helps to have some Python experience when starting with Airflow since workflows are defined in Python code. The gap in learning the basics of Airflow is not that big. Generally, getting the basic structure of an Airflow workflow up and running is easy. Let’s dig into a use case of a rocket enthusiast to see how Airflow might help him.

2.1 Collecting data from numerous sources

2.1.1 Exploring the data

2.2 Writing your first Airflow DAG

2.2.1 Tasks vs. operators

2.2.2 Running arbitrary Python code

2.3 Running a DAG in Airflow

2.3.1 Running Airflow in a Python environment

2.3.2 Running Airflow in Docker containers

2.3.3 Inspecting the Airflow UI

2.4 Running at regular intervals

2.5 Handling failing tasks

Summary