A Kubernetes pod is the smallest deployable unit in Kubernetes. It contains one or more containers that run together on a single worker node while sharing networking and storage context. In Airflow, each task can run as a container inside one pod. section 11.5.1
Pods connect task execution to Kubernetes-managed infrastructure:
- The control plane exposes an API used by kubectl or the Kubernetes Python SDK, while the scheduler assigns pods to worker nodes. section 11.5.1
- Pods can use Kubernetes-managed persistent storage through volumes and storage claims rather than relying on the Airflow worker's local storage. section 11.5.1 section 11.5.5
- Task pods are ephemeral resources, so failed or pending pods require operational inspection rather than assuming that a worker-local process is stuck. section 11.5.4
Deploying the supporting resources
Create an Airflow namespace, apply a PersistentVolume and PersistentVolumeClaim, then deploy the MovieLens API and service. Use `kubectl --namespace airflow get pods` to confirm that the API pod reaches `Running`; port-forward the service to test it locally. section 11.5.2
Diagnosing a pending task pod
Start with Airflow logs. If a task remains pending, list the pods and run `kubectl --namespace airflow describe pod [NAME-OF-POD]`. Inspect Events for scheduling failures such as a missing `data-volume` claim, apply the storage YAML, and check the pod events again. section 11.5.4
Figure 11.10 High-level overview of Kubernetes
Use the Kubernetes architecture overview to relate the control plane, scheduler, worker nodes, and pods. section 11.5.1
