DockerOperator

A pipeline may need to fetch data, run a processing script, and publish results, with different software required for each step. DockerOperator lets an Airflow DAG schedule each step as a container task: the worker starts a specified image with a command and waits for it to finish. The task’s behavior comes from that command and the software packaged in the image. section 11.3.1 section 11.4.1

DockerOperator is provided by apache-airflow-providers-docker and wraps the Python Docker client. Its arguments specify how to start the container; Airflow handles startup, logs, and, when configured, removal. On execution, the worker contacts Docker, which can fetch the image from a registry, mount storage, and run the command. section 8.3.3 section 11.4.1

rank_movies = DockerOperator(
       task_id="rank_movies",
       image="manning-airflow/movielens-rank",   #1
       command=[                     #2
           "rank_movies.py",
           "--input_path",
           "/data/ratings/{{data_interval_start | ds}}.json",
           "--output_path",
           "/data/rankings/{{data_interval_start | ds}}.csv",
       ],
       volumes=["/tmp/airflow/data:/data"],         #3
   )

The listing shows the image, templated command arguments, and host-directory mount together in one DockerOperator task. section 11.4.1

For a multi-task DAG, give each container the arguments it needs and arrange dependencies as usual. A ratings-fetch task can write a dated file to a mounted volume; a downstream ranking task reads that file through the same mount. The mount matters because a container’s own filesystem does not persist beyond its task. A network setting may also be needed when a container must reach an API on a particular Docker network; the networking detail is unnecessary for an API reachable over the internet. section 11.4.3

Plan the image lifecycle as well as the DAG:

• Package each task’s software and dependencies in an image, build it, and push it to a registry before referencing it from a DockerOperator task. section 11.4.4
• Configure commands, environment, storage, and connectivity for each task. Keep task state outside the container and avoid leftover state so reruns remain idempotent. section 8.3.3 section 11.4.3
• Test images separately from DAG orchestration: image development can have its own lifecycle, while workers pull images and run them with the DAG’s arguments. section 11.4.4

The same pattern can call a CLI in a daily recipe-ingestion pipeline: a preprocess command receives a templated MinIO path for that day’s data. DockerOperator is therefore useful when task logic belongs in an image rather than in the DAG, though it adds image-building and distribution work. section 14.5 section 11.4.4