Part 4 Airflow in production
At this point, you should be well on your way to mastering Airflow—able to write complex pipelines. So far, we’ve focused on using Airflow on a local system, either natively or using container technologies such as Docker. A common question is how to run and use Airflow in production settings. Part 4 focuses entirely on running Airflow in production, including topics such as designing architectures for Airflow deployments, securing Airflow, and choosing deployment options.
First, in chapter 15, we describe how to operate Airflow in production, touching on topics such as architectures for scaling Airflow, monitoring, logging, and alerting. Next, chapter 16 focuses specifically on securing Airflow to avoid unwanted access and minimizing the impact of security breaches. Finally, chapter 17 runs through options for deploying Airflow using managed services or deploying on your own Kubernetes cluster.
After completing part 4, you should have a good idea of how to operate and deploy Airflow and which implementation details to consider for a robust, secure deployment.