Suppose several Airflow DAGs need the same custom hook, operator, and sensor. Keeping those components under the dags directory makes them importable, but complicates sharing them with other projects and testing them separately. A Python package puts the reusable code in an installable form, separate from the DAGs. section 9.6
A basic setuptools package can be assembled in a few steps:
- Create a project directory with a src directory inside it, then create the package directory under src. section 9.6.1
- Put the hook, operator, and sensor implementations in their respective Python files. Add an __init__.py file to make the package explicit, although PEP 420 no longer requires it. section 9.6.1
- Add setup.py. Its setuptools.setup call supplies the installed name, version, dependencies, and the packages to find under src. section 9.6.1
Listing 9.35 Example setup.py file
#!/usr/bin/env python
import setuptools
requirements = ["apache-airflow", "requests"] #1
setuptools.setup(
name="airflow_movielens", #2
version="0.1.0", #2
description="Hooks, sensors and operators for the Movielens API.", #2
author="Anonymous", #3
author_email="anonymous@example.com", #3
install_requires=requirements, #4
packages=setuptools.find_packages("src"), #5
package_dir={"": "src"}, #5
url="https://github.com/example-repo/airflow_movielens", #6
license="MIT license", #7
)
The example setup.py declares apache-airflow and requests as dependencies and points setuptools at the src-based package layout. section 9.6.1
For distribution, install the package from a GitHub repository, a pip feed such as PyPI or a private feed, or a file location accessible to the Airflow environment. Separating package code from DAGs also supports a dedicated CI/CD process. If a custom operator extends an existing integration, contributing it to an existing Airflow provider is an alternative; creating a new provider takes more work. section 9.6 section 9.6.3
A package behind a DockerOperator task
A recipe-processing DAG uses DockerOperator tasks backed by a Docker image containing a Python package with a CLI. The preprocess_recipes task passes the preprocess command and a date-templated MinIO path to run the package’s preprocessing logic. section 14.5
