chapter seventeen
17 Airflow deployment options
This chapter covers
- Vendor-managed services for Airflow
- Rolling your own deployment in a Kubernetes cluster as alternative to using a managed service
- Deployment options when deploying in a Kubernetes cluster
Up to now, all our examples have been running on Airflow deployed on your local machine using Docker Compose. However, this is hardly a production-ready setup for running your pipelines outside of a development setting.
Here, we’ll explore several options for deploying Airflow in production. We’ll start out by exploring the vendor-managed solutions available and discuss criteria for using a vendor-managed solution instead of rolling your own. Afterwards, we’ll discuss how you can deploy Airflow yourself on Kubernetes. In this process, we’ll guide you through some of the choices you can make when rolling your own deployment.