12 Operating Airflow in production


This chapter covers

  • Dissecting the Airflow scheduler
  • Configuring Airflow to scale horizontally using different executors
  • Monitoring the status and performance of Airflow visually
  • Sending out alerts in case of task failures

In most of the previous chapters, we focused on various parts of Airflow from a programmer’s perspective. In this chapter, we aim to explore Airflow from an operations perspective. A general understanding of concepts such as (distributed) software architecture, logging, monitoring, and alerting is assumed. However, no specific technology is required.

12.1 Airflow architectures

12.1.1 Which executor is right for me?

12.1.2 Configuring a metastore for Airflow

12.1.3 A closer look at the scheduler

12.2 Installing each executor

12.2.1 Setting up the SequentialExecutor

12.2.2 Setting up the LocalExecutor

12.2.3 Setting up the CeleryExecutor

12.2.4 Setting up the KubernetesExecutor

12.3 Capturing logs of all Airflow processes