Suppose a data pipeline has several tasks ready to run at once, but its workload still fits on one machine. LocalExecutor lets Airflow run those tasks concurrently without setting up workers across multiple machines. It is straightforward to install and suits simple, small-scale production deployments; it does not distribute task execution across hosts. section 15.2.2 section 15.10
The scheduler queues tasks in a Python first-in, first-out queue. Worker subprocesses read from it, with each subprocess executing one task. These subprocesses run on the scheduler’s machine, even if other Airflow components run elsewhere. LocalExecutor therefore offers local parallelism, not a way past one machine’s resource limits. section 15.2.2 section 15.2.3
Configure and check a local executor
Set AIRFLOW__CORE__EXECUTOR=LocalExecutor and provide a MySQL or PostgreSQL database. AIRFLOW__CORE__PARALLELISM controls the scheduler’s maximum number of task subprocesses; its default is 32 and its minimum is 1. Pools, AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG, and AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG can restrict concurrency further. Run a DAG to check the installation: a task reaching the running state has been scheduled, queued, and picked up by the executor. section 15.2.3
For an individual task, execution starts a process that runs airflow tasks run with the DAG ID, task ID, and execution date. Airflow records the running state and heartbeats in its metastore; an exit code of zero marks the finished task successful, while a nonzero code marks it failed. section 15.4.1
Table 15.1 Overview of Airflow executors
| Executor |
Distributed |
Ease of installation |
Good fit when |
|---|---|---|---|
|
|
No |
Easy |
Running on a single machine is good enough. |
|
|
Yes |
Moderate |
You need to scale out over multiple machines. |
|
|
Yes |
Complex |
You need to scale out over multiple machines, are familiar with Kubernetes, and prefer a containerized setup. |
|
|
Yes |
Moderate |
You need an executor to run outside the Airflow’s instance environment (e.g., on premises while Airflow is in the cloud). |
Compare LocalExecutor’s easy, single-machine setup with CeleryExecutor and KubernetesExecutor when deciding whether a workload needs execution across multiple machines. section 15.2.2
