An Airflow task may need libraries that other tasks—or the Airflow workers—do not. A container image packages the software and dependencies needed to run that task. Docker creates a container from the image when the task runs; the image is not the running container. section 11.3.2 section 11.2.2
Start with a Dockerfile. FROM selects a base image; COPY adds application files; RUN installs dependencies or prepares executables. ENTRYPOINT can select the application, while CMD supplies a default argument. Build with docker build --tag name ., where the final dot makes the current directory the build context. Test the result with docker run before referencing it from a DAG. section 11.2.3
Listing 11.3 Dockerfile for fetching weather from the wttr.in API
FROM python:3.12-slim #1 COPY requirements.txt /tmp/requirements.txt #2 RUN pip install -r /tmp/requirements.txt COPY scripts/fetch_weather.py /usr/local/bin/fetch-weather #3 RUN chmod +x /usr/local/bin/fetch-weather ENTRYPOINT [ "/usr/local/bin/fetch-weather" ] #4 CMD [ "--help" ] #5
The weather-command Dockerfile starts from Python 3.12, installs requirements, copies the script, and sets its executable and default help command. section 11.2.3
For an Airflow task, first identify its software requirements and give its code a command-line interface. Build an image containing that code, then configure a DockerOperator to run the image with task-specific arguments. In a typical deployment, a CI/CD process builds and pushes the image to a registry; an Airflow worker pulls it and asks its Docker daemon to run the container. section 11.4.2 section 11.4.4
Fetching ratings for a recommender
A ratings-fetching task uses a Python CLI that accepts dates, an output path, API credentials, and a batch size. Its image starts from python:3.12-slim, installs click and requests, and copies in the script. Running the built image with fetch-ratings --help checks that the command is available before the DAG uses it. section 11.4.2
Separate task images let tasks carry different dependencies and be tested apart from DAG orchestration, but each image must be built and maintained; related tasks can sometimes share one. Images also serve a different purpose in Airflow deployment: a custom Airflow image can contain DAG files or Python libraries. Baking in DAGs makes code versions explicit but requires a new build and deployment for changes. Installing every library in one shared worker image can instead create dependency conflicts; KubernetesExecutor tasks can use different images. section 11.3.2 section 17.2.7 section 17.2.8 section 17.2.9
