13 Airflow in the Clouds
This chapter covers:
- Designing deployment strategies for several cloud platforms (Amazon AWS, Microsoft Azure and Google Cloud Platform).
- Using cloud-specific operators and hooks to integrate with different services available in the respective cloud platforms.
- Brief introduction to managed (cloud) services, which can provide an easier approach for managing Airflow deployments than rolling your own solution.
In this chapter we’ll dive into how Airflow can be used on several major cloud platforms (Amazon AWS, Microsoft Azure and Google Cloud Platform). First, we’ll do a short recap of the different parts of an Airflow deployment to get an overview of the involved components. Next, for each of the three clouds (Amazon AWS, Microsoft Azure and Google Cloud Platform), we’ll start designing several deployment strategies for each cloud by mapping the different Airflow components to appropriate cloud services. Afterwards, we’ll discuss and demonstrate how cloud-specific operators can be used in Airflow to leverage other cloud services from within DAGs. Finally, we’ll close off with a short overview of some managed cloud services, which provide an easier approach for rolling out Airflow deployments without having to manage all the underlying cloud services ourselves.