A containerized task runs an Airflow task inside an isolated container rather than directly in the Airflow worker environment. This approach packages task dependencies with the task and gives different tasks a consistent execution interface. section 11.3.2 section 11.6
The practical pattern is to build an image for each task, then run that image with DockerOperator. Tasks can use different images and commands while sharing the same operator configuration pattern. section 11.3.2 section 11.4.3
Containerized tasks are useful because they provide:
- Isolated dependencies: each image contains the exact packages required by its task, avoiding conflicts and worker-environment installation. section 11.3.2
- A uniform interface: developers learn one operator and debug a common integration point across otherwise different tasks. section 11.3.2
- Improved testability: images can be developed, maintained, and tested independently from the Airflow DAG. section 11.3.2
- Flexible execution: Docker runs tasks locally or through DockerOperator, while KubernetesPodOperator runs them in Kubernetes pods for cluster scaling. section 11.6
MovieLens ratings and ranking pipeline
A fetch_ratings DockerOperator runs the fetch-ratings command in the manning-airflow/movielens-fetch image. It passes API credentials and date arguments, writes ratings to /data, and mounts a host path so results outlive the container. A dependent rank_movies task runs rank-movies in the manning-airflow/movielens-rank image, reads the mounted ratings file, writes rankings, and follows the dependency fetch_ratings >> rank_movies. section 11.4.3
Figure 11.6 Managing dependencies across different tasks using containers
Use the dependency-management figure to compare how separate task images isolate requirements across a workflow. section 11.3.2
Container filesystems disappear when tasks finish, so related tasks need an explicit exchange mechanism such as a volume mount. Network configuration may also be required when a container accesses another service on a Docker network. section 11.4.3
