2 Anatomy of an Airflow DAG

 

This chapter covers

  • Running Airflow on your own machine
  • Writing and running your first workflow
  • Examining the first view at the Airflow interface
  • Handling failed tasks in Airflow

In the previous chapter, we learned why working with data and the many tools in the data landscape are not easy tasks. In this chapter, we get started with Airflow and check out an example workflow that uses basic building blocks found in many workflows.

It helps to have some Python experience when starting with Airflow since workflows are defined in Python code. The gap in learning the basics of Airflow is not that big. Generally, getting the basic structure of an Airflow workflow up and running is an easy task. Let’s dig into a use case of a rocket enthusiast to see how Airflow might help him.

2.1       Collecting data from numerous sources

2.1.1   Exploring the data

2.2       Writing your first Airflow DAG

2.2.1   Tasks vs operators

2.2.2   Running arbitrary Python code

2.3       Running a DAG in Airflow

2.4       Running at regular intervals

2.5       Handling failing tasks

2.6       Summary