9 Testing

This chapter covers

  • Testing Airflow tasks in a CI/CD pipeline
  • Structuring a project for testing with pytest
  • Mimicking a DAG run to test tasks that apply templating
  • Faking external system events with mocking
  • Testing behavior in external systems with containers

In all previous chapters, we focused on various parts of developing Airflow. So how do you ensure the code you’ve written is valid before deploying it into a production system? Testing is an integral part of software development, and nobody wants to write code, take it through a deployment process, and keep their fingers crossed for all to be okay. Such a way of development is obviously inefficient and provides no guarantees on the correct functioning of the software, both in valid and invalid situations.

This chapter will dive into the gray area of testing Airflow, which is often regarded as a tricky subject. This is because of Airflow’s nature of communicating with many external systems and the fact that it’s an orchestration system, which starts and stops tasks performing logic, while Airflow itself (often) does not perform any logic.

9.1 Getting started with testing

9.1.1 Integrity testing all DAGs

9.1.2 Setting up a CI/CD pipeline

9.1.3 Writing unit tests

9.1.4 Pytest project structure

9.1.5 Testing with files on disk

9.2 Working with DAGs and task context in tests

9.2.1 Working with external systems

9.3 Using tests for development

9.3.1 Testing complete DAGs

9.4 Emulate production environments with Whirl

9.5 Create DTAP environments