When Airflow tasks outgrow one machine, KubernetesExecutor can distribute their execution across a Kubernetes cluster. Each task runs in a Kubernetes pod, giving workloads strong isolation. This is a good fit when Kubernetes is already part of the environment, but it brings more setup complexity and overhead than simpler executors. section 15.2.2 section 15.10 section 15.2.3
Choose an executor around the workload and the infrastructure you can operate:
- LocalExecutor runs parallel tasks on one machine; it is the simpler choice when that capacity is enough. section 15.2.2
- CeleryExecutor also distributes tasks across machines. Even on Kubernetes, its lower overhead may make it preferable for some workloads. section 15.2.2 section 15.10
- KubernetesExecutor suits distributed, containerized tasks when isolation matters and a Kubernetes cluster is available to configure and operate. section 15.2.2 section 15.10
Select it with AIRFLOW__CORE__EXECUTOR=KubernetesExecutor. The cluster must be set up and configured for Airflow. To check an executor installation, run a DAG: a task reaching the running state has passed through scheduling and queuing and has been picked up by an executor. The API server, scheduler, and database need not run in Kubernetes, although colocating them there can simplify this setup. section 15.2.1 section 15.2.3
Figure 15.5 With the KubernetesExecutor, all tasks run in a pod in Kubernetes. Although it isn’t necessary to run the API server, scheduler, and database in Kubernetes, it’s sensible to run it there when using the KubernetesExecutor to reduce the complexity of the setup.
The diagram places task execution in Kubernetes pods and distinguishes those pods from Airflow’s other components. section 15.2.3
Different dependencies for different tasks
A Helm deployment can configure CeleryExecutor and KubernetesExecutor together. In one example DAG, a Celery task fails because its default image lacks TensorFlow; two KubernetesExecutor tasks succeed using images with the required dependency, including a task-specific image override. While they run, kubectl -n airflow get pods shows the Kubernetes task pods. This makes per-task image choice useful when tasks need different dependencies. section 17.2.9
Multiple executors can be configured together so tasks use different execution environments. Reserve the added Kubernetes setup and overhead for workloads that benefit from its isolation or image flexibility, rather than assuming every task needs it. section 15.2.1 section 17.2.9 section 15.10
