This chapter will dive further into how to deploy and integrate Airflow with cloud services in the Microsoft Azure cloud. First, we’ll start designing an Airflow deployment by mapping the different components of Airflow to Azure services. Then we’ll explore some of the hooks and operators that Airflow provides for integrating with several key Azure services. Finally, we’ll show how to use these Azure-specific operators and hooks to implement a use case for generating movie recommendations.
In chapter 15, we described the different components comprising an Airflow deployment. In this section, we’ll design a few deployment patterns for Azure by mapping them to specific Azure cloud services. This should give you a good idea of the process involved in designing an Airflow deployment for Azure and provide a good starting point for implementing one.
Let’s start with the Airflow webserver and scheduler components. One of the easiest approaches for running these components is to use Azure’s managed container services, such as Azure Container Instances (ACI) or Azure Kubernetes Service (AKS). However, for the webserver, we also have another option: the Azure App Service.