Welcome
Thank you for purchasing the MEAP for Data Pipelines with Apache Airflow. We hope the book in its current state is already valuable to you and aim to make the final version of the book even better using your feedback!
This book aims to provide an in-depth introduction for data-savvy professionals to Airflow, a tool/framework that helps you develop complex data pipelines consisting of many tasks and (possibly) spanning different technologies. When we started writing our first Airflow pipelines it was a relief to see a simple Python script gluing together various tasks and handling the complex logic of dependencies, retries, logging, and such.
The Apache Airflow framework holds many possible options for writing, running, and monitoring pipelines. While this gives a lot of freedom to define pipelines in whichever way you like, it also results in no single good or the best way to do so. This book aims to provide a guide to the Airflow framework from start to end, together with best practices and lessons learned from our experience of using Apache Airflow.