Suppose an Airflow pipeline must fetch ratings, rank movies, and post results. Each step needs different software. A container lets the step run with its dependencies packaged in an image, rather than requiring every dependency in the Airflow environment. A container-based operator starts the container and waits for its command to finish. section 11.1.3 section 11.3.1
Containers isolate applications and dependencies using host-OS kernel functionality. Unlike a virtual machine, each container does not need its own guest operating system. A container engine such as Docker manages images and running containers; an image supplies the software, while a container runs a command from it. section 11.2.1 section 11.2.2
Listing 11.1 Running a Docker container
$ docker run debian:buster-slim echo Hello, world!
Run `docker run debian:buster-slim echo Hello, world!` with Docker installed and running. Docker pulls the image if needed, creates a container, executes the command inside it, and streams the output to your terminal. section 11.2.2
Containers have three useful roles around an Airflow pipeline:
- Run Airflow itself: Docker Compose defines the database, workers, scheduler, and other services in a multicontainer setup. `docker compose up` starts the setup; `docker ps` shows running containers. section 2.3.2
- Run a task: `DockerOperator` starts an image with the command, environment, and network settings the task needs. Airflow starts the container, retrieves logs, and can remove it afterward; avoid leaving state behind so reruns remain idempotent. section 11.3.1 section 8.3.3
- Test an external dependency: a PostgreSQL container gives an operator test a real database when the production database is unavailable, so PostgreSQL hook calls need not be mocked. section 10.2
Listing 14.1 Calling the CLI command to preprocess the recipe from our task
import pendulum
from airflow.providers.docker.operators.docker import DockerOperator
from airflow.timetables.trigger import CronTriggerTimetable
from airflow import DAG
ENVIRONMENT = {
# All our environment variables
}
docker_kwargs= { #1
"image":" vectorvault_cli:latest",
"environment":ENVIRONMENT,
"docker_url":"tcp://docker-socket-proxy:2375",
"network_mode":"chapter14_default",
"environment":ENVIRONMENT,
"auto_remove":"success",
"tty":True,
}
with DAG(
dag_id=" 01_ingestion",
schedule=CronTriggerTimetable("@daily", timezone="UTC"),
start_date=pendulum.today("UTC"),
):
...
preprocess_recipes = DockerOperator(
task_id="preprocess_recipes",
command=[
"preprocess", #2
"s3://data/{{data_interval_start | ds}}", #3
],
**docker_kwargs #4
)
The recipe-preprocessing task uses `DockerOperator` to run a CLI command from a shared image. Its command includes a templated daily MinIO path; container configuration supplies the Docker connection and network. section 14.5
Testing an operator against PostgreSQL
Use `pytest-docker-tools` to fetch `postgres:16-alpine` and start a container with test credentials, an exposed port, and a mounted SQL initialization file. Point the MovieLens-to-PostgreSQL operator at that database, run it, and check the written results. This tests the interaction with a real database without connecting to production; Docker must be available on the test machine. section 10.2
Choose the container’s role deliberately. Docker Compose provides a reproducible Airflow environment; `DockerOperator` keeps task-specific processing outside Airflow’s own dependencies; a short-lived test container supplies a realistic external system. For heavier processing, a containerized task can run on compute separate from the Airflow cluster, leaving Airflow to orchestrate. section 2.3.2 section 14.5
