A sensor waiting for an external file or dataset can spend most of its run time doing nothing. In its default poke mode, it still occupies a task slot between checks; enough waiting sensors can leave no slots for other work. A deferrable operator avoids holding a worker slot while it waits: it defers to a trigger running in Airflow’s triggerer. section 7.1.2
A trigger is an asynchronous Python component that waits for a condition on behalf of a deferred task. The triggerer can handle many triggers concurrently. Once the wait is over, the task returns to a worker to finish. Deferrable operators require a triggerer process alongside the scheduler; simply choosing a deferrable operator is not enough if that process is absent. section 7.1.2 section 9.5.2
For a custom deferrable sensor, separate the task’s handoff from the trigger’s wait:
- Implement the sensor as a BaseOperator whose execute method calls defer with a trigger, a completion method name, and a timeout. The completion method runs when the task resumes. section 9.5.2
- Implement the trigger’s serialize method with its class location and parameters so Airflow can reconstruct it after a pause or move. section 9.5.2
- Put the waiting logic in the trigger’s asynchronous run method, rather than in a sensor poke method. section 9.5.2
Wait for ratings before fetching them
A daily DAG can defer a ratings sensor until records for its data interval are available, then run a ratings-fetch task. The sensor passes its connection, interval, and polling interval to a custom trigger; a 60-minute execution timeout fails the task if the wait runs too long. For a ready-made option, FileSensorAsync provides a deferrable file sensor. section 9.5.2 section 7.1.2
Figure 9.7 Deferrable sensor execution sequence. Task execution begins in the scheduler. When the task is ready to run, the execution of the sensor moves to a worker (1). The worker passes the execution of the deferred task to the triggerer (2), using the defer method. The triggerer spends most of the run time waiting for the external task to complete. When it finds new records, the triggerer returns the execution to the worker (3). The worker runs any remaining commands defined in the execute_complete method. Finally, the scheduler (4) marks the task as successful and proceeds with the execution of any downstream tasks.
The execution sequence shows the worker handing a deferred task to the triggerer, then resuming it on a worker when the wait ends. section 9.5.2
A sensor in reschedule mode is another way to release a slot between checks. Deferral instead moves the wait to the triggerer; choose it when a deferrable operator is available and the triggerer is running. Python and TaskFlow API Python operators cannot themselves be deferred. section 7.1.2
