1 Meet Apache Airflow
This chapter covers
- Representing data pipelines in workflows as graphs of tasks
- How Airflow fits into the ecosystem of workflow managers
- Determining if Airflow is a good fit for you
Enterprises are continuously becoming more data-driven and are developing data pipelines as part of their daily business. Data volumes involved in these business processes have increased substantially over the years, from megabytes per day to gigabytes per minute. Though handling this data deluge may seem like a considerable challenge, these increasing data volumes can be managed with the appropriate tooling.
Apache Airflow is one of those tools. Airflow tackles large-scale data processing by splitting it into smaller, time-defined, and more manageable chunks of data known as batches. We focus on building data pipelines with Apache Airflow. However, this doesn’t mean Apache Airflow can’t be used to orchestrate and schedule other workloads. If you can communicate with a system or tool using Python, you can potentially manage it with Airflow. In fact, one of Airflow’s key features is that it enables you to easily build scheduled data pipelines using Python, while also providing many building blocks that allow you to stitch together the many different technologies encountered in modern technological landscapes.