When a DAG’s tasks outgrow one machine, CeleryExecutor lets Airflow distribute their execution across machines. Airflow queues tasks through Celery; Celery workers read the queue and execute them. This resembles LocalExecutor’s queue-and-worker model from a task’s perspective, but its components can run on different virtual machines. section 15.2.2
Figure 15.3 In the CeleryExecutor, tasks are divided among multiple machines running Celery workers. The workers wait for tasks to arrive in a queue.
The diagram shows tasks divided among machines whose Celery workers wait for queued work. section 15.2.3
To run and assess the setup:
- Set AIRFLOW__CORE__EXECUTOR to CeleryExecutor, provide a Celery queueing mechanism, and start workers with airflow celery worker. Stable queueing options named here are RabbitMQ, Redis, and AWS Simple Queue Service. section 15.2.1 section 15.2.2
- Run a DAG to check the installation. A task reaching running state has been scheduled, queued, and picked up by an executor; Flower can show Celery worker status. section 15.2.3 section 15.2.2
- Size workers and task limits together. AIRFLOW__CELERY__WORKER_CONCURRENCY limits processes per worker, while global parallelism, per-DAG limits, and pools can restrict how many tasks run. Budget at least 200 MB of RAM per worker process as a starting baseline, then account for the most resource-intensive tasks running together. section 15.4.4
A task succeeds on one executor but fails on another
In a deployment combining CeleryExecutor and KubernetesExecutor, a Celery task fails because TensorFlow is missing from its default worker image. Two Kubernetes tasks succeed using images that contain the dependency. The result makes the operational distinction concrete: choosing an executor also means checking the environment in which its tasks run. section 17.2.9
Choose CeleryExecutor when spreading work across machines is worth its additional setup. LocalExecutor is simpler but stays on one machine; KubernetesExecutor also scales out and offers greater isolation, with more overhead. A deployment can combine executors and select one for particular tasks when their needs differ. section 15.10 section 15.2.1
