17 Airflow on Azure
This chapter covers:
- Designing a deployment strategy for Azure using the Azure App Service, Azure Container Instances, Azure File/Blob storage and Azure SQL services.
- An overview of several Azure-specific hooks and operators that allow you to integrate with commonly used Azure services.
- Demonstrating how to use Azure-specific hooks and operators to build a simple serverless recommender system.
After our brief cloud introduction in Chapter 15, this chapter will dive further into how to deploy and integrate Airflow with cloud services in the Microsoft Azure cloud. First, we’ll start by designing an Airflow deployment by mapping the different components of Airflow to Azure services. Afterwards, we’ll explore some of the hooks and operators that Airflow provides for integrating with several key Azure services. Finally, we’ll show how to actually use these Azure-specific operators and hooks to implement an actual use case for generating movie recommendations.
In Chapter 15, we described the different components comprising an Airflow deployment. In this section, we’ll design a few deployment patterns for Azure by mapping these different components to specific Azure cloud services. This should hopefully give you a good idea of the process involved in designing an Airflow deployment for Azure and provide a good starting point for actually implementing one.