chapter ten
10 Testing
This chapter covers
- Testing Airflow tasks in a CI/CD pipeline
- Structuring a project for testing with
pytest - Testing individual operators
- Faking external system events with mocking
- Using containers to test behavior in external systems
Up to now, we’ve focused on building data pipelines with Airflow. But how do you ensure that the code you’ve written is valid before deploying it into a production system? As in any software development process, testing your DAGs is a crucial step toward ensuring that they function correctly in both normal situations (the “happy” flow) and edge cases.
In this chapter, we explore testing in Airflow. This topic is often regarded as a tricky one due to the interconnectedness of data pipelines with external systems, which can make pipelines difficult to test. That’s no excuse to skip writing tests, however. We’ll show you how to write effective tests for your pipelines, resulting in more reliable systems.