Use the defer method when an Airflow task must wait for external data without keeping its execution logic on a worker. The worker transfers the waiting task to the triggerer, which monitors the condition and later returns execution to the worker. section 9.5.2
A deferred operator coordinates waiting and completion through these elements:
- Set a timeout so prolonged waiting raises an exception and fails the task. section 9.5.2
Waiting for MovieLens ratings
A custom sensor calls self.defer(trigger=MovielensTrigger(...), method_name="execute_complete", timeout=self._timeout). After ratings appear, execute_complete logs that they are available; a downstream fetch task can then proceed.
