A pool protects a shared resource when parallel Airflow tasks could overwhelm it. Assign related tasks to a named pool, then set its fixed slot count to the concurrency the resource can support. section 12.4
Use pools to coordinate tasks competing for:
- Databases that should not receive too many simultaneous queries. section 12.4
- GPU systems or Spark clusters with limited processing capacity. section 12.4
- High-resource tasks that need stricter limits than ordinary tasks. section 15.4.4
Assign a task to a pool
Create a pool through Airflow's Admin > Pools interface, give it a name and description, and choose its slot count. Then assign that pool when defining a task:\n\nPythonOperator(\n task_id="my_task",\n ...,\n pool="my_resource_pool"\n)\n\nWith 10 slots, the pool permits 10 tasks to access the resource simultaneously. When no slot is available, the scheduler postpones the task until one is released. section 12.4
