Historical backfills can create hundreds of parallel DAG runs, crowding out other workflows or overloading external systems. Set a DAG's max_active_runs to cap how many runs may be active simultaneously. section 3.6
When the limit is reached, Airflow does not create new DAG runs until earlier runs complete. An unset DAG-level value inherits core.max_active_runs_per_dag, which defaults to 16. section 3.6 section 15.4.4
Limit concurrent backfill runs
dag = DAG('my_dag', max_active_tasks=10, max_active_runs=2) This configuration allows at most two active runs of my_dag, with at most ten task instances running in parallel across those runs. section 3.6
Use the setting alongside other concurrency controls:
- Use max_active_tasks to limit parallel task instances within a DAG. section 3.6
- Use pools to constrain particular task groups, especially resource-intensive work. section 3.6 section 15.4.4
- Account for global parallelism and Celery worker concurrency when estimating actual capacity. section 15.4.4
Table 15.2 Overview of Airflow configurations related to running number of tasks
| Configuration item |
Default value |
Description |
|
|
|
The maximum number of tasks to be in the queued or running state per DAG |
|
|
|
The maximum number of parallel DAG runs per DAG |
|
|
|
The maximum number of task instances to run in parallel globally |
|
|
|
The maximum number of tasks per Celery worker |
Compare max_active_runs_per_dag with DAG task limits, global parallelism, and Celery worker concurrency. section 15.4.4
