A task can be ready to run without yet having a place to run. Airflow’s scheduler checks dependencies and available slots, then queues an eligible task instance. The executor takes responsibility for the queued task and starts it on a worker. Choosing an executor determines whether that work stays on one machine, spreads across machines, or runs in another environment. section 15.4.1 section 15.2
Choose according to where tasks must run and what the deployment can support:
- LocalExecutor runs parallel worker subprocesses on one machine. It is easy to set up and fits a small, simple deployment, but cannot distribute tasks across machines. section 15.2.2 section 15.2.3
- CeleryExecutor distributes tasks to Celery workers through a queue. It suits workloads that need multiple machines, at the cost of setting up a queuing mechanism. section 15.2.2
- KubernetesExecutor runs tasks in Kubernetes pods. It offers containerized task isolation and scale-out, but requires a Kubernetes cluster and a more complex setup. section 15.2.2 section 15.2.3
- EdgeExecutor can run workers outside the main Airflow environment—for example, on premises when Airflow runs in the cloud. It is experimental. section 15.2.2 section 15.10
Configure and verify an executor
Set `AIRFLOW__CORE__EXECUTOR=LocalExecutor` for a single-machine setup; in Airflow 3, LocalExecutor is already the default. Its subprocess limit is controlled by `AIRFLOW__CORE__PARALLELISM`, whose default is 32. Run a DAG to check the setup: a task reaching `running` has passed through scheduling, queuing, and executor pickup. Configuration can also be placed in `airflow.cfg`, although environment variables take precedence. section 15.2.3 section 15.2.1
Route tasks to different executors
A deployment can configure `CeleryExecutor,KubernetesExecutor` together and select an executor on a task, such as `BashOperator(task_id="hello_world", executor="LocalExecutor", bash_command="echo 'hello world!'")` when LocalExecutor is configured. In one mixed Celery/Kubernetes deployment, a task using Celery’s default image fails because TensorFlow is missing, while Kubernetes tasks using images with that dependency succeed. Executor choice can therefore affect both placement and which task dependencies are available. section 15.2.1 section 17.2.9
Table 15.1 Overview of Airflow executors
| Executor |
Distributed |
Ease of installation |
Good fit when |
|---|---|---|---|
|
|
No |
Easy |
Running on a single machine is good enough. |
|
|
Yes |
Moderate |
You need to scale out over multiple machines. |
|
|
Yes |
Complex |
You need to scale out over multiple machines, are familiar with Kubernetes, and prefer a containerized setup. |
|
|
Yes |
Moderate |
You need an executor to run outside the Airflow’s instance environment (e.g., on premises while Airflow is in the cloud). |
Compare executor types by distribution, installation effort, and deployment fit before committing to an execution mode. section 15.2.2
