A task slot is a unit of worker capacity allocated to a running Airflow task. A task keeps its slot while it executes or waits, so waiting tasks can prevent runnable tasks from starting. section 7.1.2
Sensor deadlock
In a supermarket couponing DAG, four sensors can occupy 16 tasks while polling. If the DAG reaches its task limit, two new tasks cannot run; if the global limit is reached, the whole Airflow system stalls. section 7.1.2
Reduce unnecessary slot occupancy with these approaches:
- Use sensor mode="reschedule" to release the slot after each poke and reacquire it for subsequent work. section 7.1.2
- Use deferrable operators to hand waiting work to the triggerer, freeing the worker slot until completion needs worker execution again. section 7.1.2
Figure 7.10 When you use a standard operator, during the entire task duration, a worker slot is allocated, blocking other tasks from using that slot. A deferrable operator, however, frees the worker slot while the job is running and being polled for its status, enabling Airflow to use that worker slot for other tasks while the polling takes place. When the polling phase is finished, the worker slot can be reallocated to the deferrable operator task to complete the work.
Compare continuous worker-slot allocation with the release and reallocation behavior of deferrable operators. section 7.1.2
