Asynchronous programming lets a task begin and then proceed without continuously waiting for it to finish. In Airflow, this is useful when tasks spend much of their time waiting for external systems, such as a Spark job. section 9.5.1
Airflow’s triggerer handles this waiting work. A deferrable task moves from a worker to the triggerer, which monitors its external condition asynchronously and leaves the worker available for other tasks. section 9.5.1 section 1.2.3
The execution model separates active work from waiting:
- Workers perform tasks that are ready to run. section 1.2.3
- The triggerer monitors deferred tasks and external completion conditions. section 9.5.1
- The scheduler checks dependencies and can defer tasks that depend on external resources. section 1.2.3
- Python’s asyncio functionality enables one component to manage many waiting tasks concurrently. section 9.5.1
Waiting for a Spark job
With only ten workers, ten long-running jobs could occupy the workers in batches. Deferring those jobs lets the triggerer monitor their Spark responses concurrently, allowing workers to handle tasks that are ready to run. section 9.5.1
Figure 9.6 In the standard (nondeferrable) method of running tasks, each task must be assigned to a worker. The deferrable approach transfers all the tasks to the triggerer.
Compare standard execution, where each task occupies a worker, with deferrable execution, where waiting tasks move to the triggerer. section 9.5.1
