Polling repeatedly checks an external condition until it becomes true or a timeout is reached. In Airflow, sensors use polling to detect data availability without delaying an entire workflow unnecessarily. section 7.1
Airflow polling behaves as follows:
- A sensor checks a condition, such as whether a file exists. section 7.1
- A false result causes the sensor to wait and try again after the configured interval. section 7.1
- A true result moves the task to Success; continued failure ends when the configured timeout is reached. section 7.1
- Very short intervals can overload the target system when responses take as long as the polling interval. section 7.1
Waiting for supermarket data
Instead of starting a workflow at 2:00 and leaving it idle while supermarkets deliver data, a FileSensor can repeatedly check for /data/supermarket1/data.csv and begin processing when the file appears. section 7.1
Listing 7.1 A FileSensor waits for a file path to exist
from airflow.providers.standard.sensors.filesystem import FileSensor wait_for_supermarket_1 = FileSensor( task_id="wait_for_supermarket_1", filepath="/data/supermarket1/data.csv", )
Inspect the FileSensor configuration that polls for a file path. section 7.1
