A data pipeline can appear ready to deploy while a DAG fails to load, an operator returns the wrong result, or a connection call fails. Testing checks expected behavior before deployment, including ordinary inputs and edge cases. Choose the test by the failure you want to catch. section 10.0 section 10.5
Use different tests for different boundaries:
- A DAG integrity test catches basic DAG errors. section 10.5
- A unit test checks one function or operator. Run an operator’s execute() method with chosen inputs and assert its result; include invalid inputs where behavior matters. section 10.1 section 10.1.3
- An integration test checks components together. Simulate the production environment as closely as possible; a passing unit test alone does not establish that the assembled system works. section 10.1 section 10.5
Test an operator without a live Airflow system
A pytest test function can construct a BashOperator, call `task.execute(context={})`, and assert that the result is `"hello!"`. For a popularity operator, a test can instead assert the number of returned movies. A failed test may expose a setup mistake first: one example fails because the operator was constructed without its required `conn_id`. section 10.1.4 section 10.1.3
Listing 10.15 Mocking an object in a test
def test_movielenspopularityoperator(mocker):
mocker.patch.object(
MovielensHook,
"get_connection",
return_value=Connection(
conn_id="test",
login="airflow",
password="airflow",
),
)
task = MovielensPopularityOperator(
task_id="test_id",
conn_id="test",
start_date="2015-01-01",
end_date="2015-01-03",
top_n=5,
)
result = task.execute(context=None)
assert len(result) == 3
This test uses pytest-mock’s `mocker` fixture to replace a metastore connection lookup with a predefined Connection. The operator can then run without that database, and the test asserts its result. A captured mock can also verify the lookup’s arguments and call count. section 10.1.4
Run tests in CI/CD to catch issues early. During development, run a test under an IDE debugger to inspect variables at a breakpoint; `pdb` offers a command-line alternative. `airflow tasks test` can run an individual task without recording it in the metastore. section 10.5 section 10.3
