17 Airflow deployment options
This chapter covers
- Vendor-managed services for Airflow
- Rolling your own deployment in a Kubernetes cluster as alternative to using a managed service
- Deployment options when deploying in a Kubernetes cluster
Until now we have explored Airflow in a single instance setup with docker compose. In chapter 15 we learned about some of the different configuration options available to operate Airflow in production. Now, we will focus on the deployment of Airflow. We’ll start exploring the vendor-managed solutions available and discuss several criteria you should consider when weighing rolling your own deployment versus using a vendor-managed solution. Finally, we focus on deploying Airflow in Kubernetes, looking at the various components of Airflow and how these fit together in these deployments. We’ll use this breakdown to guide you through some of the choices you as a Data Engineer can make when rolling your own deployment.