Docker provides isolated environments for reproducible applications. A container can package Python libraries together with dependencies such as database drivers or compilers, without requiring a separate guest operating system. section 2.3.2 section 11.2.1
For Airflow, Docker becomes especially useful as configurations grow and pipelines depend on multiple services. Docker Compose defines those services and their configuration in YAML, including databases, workers, schedulers, and related components. section 2.3.2 section 11.0
A Docker-based Airflow setup typically involves:
- Installing Docker Engine, commonly through Docker Desktop. section 2.3.2
- Defining the base services in a Docker Compose YAML file and extending them with chapter- or project-specific overrides. section 2.3.2
- Starting the services together so Airflow can initialize its database, create its first user, and run its components. section 2.3.2
- Running selected Airflow tasks inside Docker containers when operator diversity makes deployment and maintenance difficult. section 11.0 section 11.4
Start a local Airflow environment
From the project directory, run `docker compose up`. After the services finish configuring, run `docker ps` to inspect the Airflow worker, API server, DAG processor, scheduler, triggerer, PostgreSQL, and Redis containers.
Listing 2.10 Running docker compose up to get Airflow ready to run
@user $ cd chapter02 @user $ docker compose up
Shows the command sequence for starting the Compose-managed Airflow environment. section 2.3.2
Listing 2.11 List of the Airflow services running
@user $ docker ps CONTAINER ID IMAGE NAMES 8e2d4649dfbd apache/airflow:3.1.0 chapter02-airflow-worker-1 2229516745e6 apache/airflow:3.1.0 chapter02-airflow-apiserver-1 1e9ee7e33ec6 apache/airflow:3.1.0 chapter02-airflow-dag-processor-1 1f57735aa785 apache/airflow:3.1.0 chapter02-airflow-scheduler-1 ac23f91faab2 apache/airflow:3.1.0 chapter02-airflow-triggerer-1 7cd704c8fce4 postgres:16 chapter02-postgres-1 5b867091e894 redis:7.2-bookworm chapter02-redis-1
Lists the containers created for the Airflow services and supporting infrastructure. section 2.3.2
