chapter seventeen
17 Airflow deployment options
This chapter covers
- Vendor-managed services for Airflow
- Rolling your own deployment in a Kubernetes cluster instead of using a managed service
- Deployment options when deploying in a Kubernetes cluster
Up to now, all our examples have run on Airflow deployed on your local machine using Docker Compose. This setup, however, is hardly production ready for running your pipelines outside a development setting.
In this chapter, we’ll cover several options for deploying Airflow in production. We’ll start by exploring the vendor-managed solutions available and discuss criteria for using a vendor-managed solution instead of deploying it yourself. Then we’ll discuss how you can deploy Airflow on Kubernetes. In the process, we’ll guide you through some choices you can make during such a deployment.