A data pipeline may need more than Airflow to run its tasks. An HTTP task can require the Python package requests; a MySQL task can require Python or system-level MySQL components; machine-learning code may need pandas or scikit-learn. These packages and components are software dependencies: requirements of the code that performs the work. section 11.1.2
Do not confuse software dependencies with task dependencies. A task dependency says which task must finish before another can start. In a pipeline that downloads launch data, fetches pictures, and sends a notification, Airflow expresses that ordering with `download_launches >> get_pictures >> notify`. Installing the packages needed by those tasks is a separate concern. section 2.2 section 11.1.2
Where tasks run determines where their software dependencies belong:
- When operators run tasks directly in Airflow, their dependencies must be installed in both the scheduler and worker environments. Many operators can make those environments difficult to maintain and introduce security risks. section 11.1.2
- Different DAGs or tasks can require incompatible package versions. A single Python environment cannot install multiple versions of the same package, so such conflicts must be resolved. section 11.1.2
- With containerized tasks, install each task’s required packages in its Docker image instead. Isolated images avoid conflicts between tasks and keep task dependencies out of the Airflow worker environment, but the images must be built and maintained. section 11.3.2
Choosing a boundary for two tasks
Suppose an HTTP task uses requests while a model-training task uses pandas or scikit-learn. Running them directly through Airflow operators puts their requirements into the shared Airflow environment. Running each as a containerized task lets each image carry its own requirements; related tasks may be able to share an image. section 11.1.2 section 11.3.2
Figure 11.2 Complex and conflicting dependencies between Airflow tasks or DAGs. Running many DAGs in a single environment can lead to conflicts when DAGs depend on different versions of the same (or related) packages. Python in particular doesn’t support installing different versions of the same package in the same environment. This means we have to resolve any conflicts in packages (right) by rewriting the DAGs or their dependencies to use the same package versions.
The dependency-conflict diagram shows why tasks or DAGs needing different package versions are troublesome in a shared environment. section 11.1.2
