Kubernetes is an open-source container orchestration platform for deploying, scaling, and managing containerized applications across a cluster of machines. It is useful when Docker on one machine cannot distribute work or scale across multiple machines. section 11.5 section 11.5.1
A Kubernetes deployment separates management from execution:
- The control plane exposes the API server, scheduler, and services for managing deployments and storage. section 11.5.1
- Worker nodes run the applications assigned by the scheduler. section 11.5.1
- Pods are Kubernetes’s smallest work units; an Airflow task runs as a container inside one pod. section 11.5.1
- Scheduling can account for CPU, memory, storage, and special hardware such as GPUs. section 11.5.1
Running an Airflow task with Kubernetes
Prepare a cluster and install kubectl, then create an airflow namespace. Define a persistent volume and storage claim with YAML, apply them with kubectl, and deploy the MovieLens API as a Kubernetes deployment and service. Airflow can then run containerized tasks in pods, while Kubernetes places those pods on suitable worker nodes. Unlike Docker-based execution, task resources and storage belong to the Kubernetes environment rather than the Airflow worker. section 11.5.2 section 11.5.5
Figure 11.10 High-level overview of Kubernetes
Use the architecture figure to trace the control plane, worker nodes, and containerized workloads. section 11.5.1
Kubernetes improves scalability, workload-specific resource placement, storage, and secret management, but requires maintaining a cluster. Airflow itself can also run inside the cluster. section 11.5.5
