Suppose 100 Spark jobs each take an hour. If an Airflow worker stays assigned to each job while Spark does the computation, a system with 10 workers must run the jobs in batches. The workers spend much of that time waiting rather than doing work. section 9.5.1
The triggerer handles this waiting for deferrable tasks. Instead of keeping a worker occupied, Airflow defers the task to the triggerer. The triggerer does not perform the external job; it monitors its status or completion condition. Using Python’s asyncio, one triggerer can monitor many waiting tasks concurrently. section 9.5.1
Its role alongside other Airflow components is distinct:
- The scheduler queues tasks whose schedules and dependencies permit execution; workers perform the work assigned to them. section 1.2.3
- The triggerer checks conditions for tasks that support asynchronous processing, freeing workers for tasks ready to run. section 1.2.3
- The triggerer is optional in an Airflow deployment, but required to use deferrable operators. section 15.1
Figure 9.6 In the standard (nondeferrable) method of running tasks, each task must be assigned to a worker. The deferrable approach transfers all the tasks to the triggerer.
Compare worker assignment in the standard approach with transferring waiting tasks to the triggerer in the deferrable approach. section 9.5.1
