liveBook · Manning

Max active tasks

As explained in Data Pipelines with Apache Airflow, Second Edition

Max active tasks is an Airflow DAG setting that limits the number of tasks from that DAG that may run simultaneously.

Backfilling historical data can produce many DAG runs in quick succession. If their tasks all compete to run, they can crowd out other work or overload external systems. `max_active_tasks` places a DAG-level limit on that activity. section 3.6

Set `max_active_tasks` on a DAG to cap the task instances from that DAG that run simultaneously, across its runs. If unset, it inherits `core.max_active_tasks_per_dag`, which defaults to 16. This is distinct from `max_active_runs`, which limits active DAG runs. section 3.6

dag = DAG('my_dag', max_active_tasks=10, max_active_runs=2)

With `max_active_tasks=10` and `max_active_runs=2`, this DAG can run at most ten tasks across at most two active DAG runs. section 3.6

When waiting tasks consume the limit

Suppose a daily DAG starts sensors for several supermarkets. If three sensors remain running each day, their occupied task slots accumulate: after five days, 15 slots are taken; on day six, a 16-task limit blocks two new tasks. Raising `max_active_tasks` can admit more work, but sensors in `reschedule` mode address the waiting behavior by releasing their slots after polling. section 7.1.2

Choose the limit alongside other capacity controls. Global `parallelism` can restrict tasks even when a DAG permits more; pools can limit selected resource-heavy tasks. For backfills, `max_active_runs` can also prevent new runs from being created until earlier ones finish. section 15.4.4 section 3.6

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Defines the parameter, gives its inherited default, and contrasts it with run limits; does not detail scheduler internals.
Examples
Uses a DAG capped at ten tasks across two runs and a daily sensor backlog that fills a 16-task limit.
What it assumes
Assumes basic familiarity with DAGs and tasks while defining how their task instances share a concurrency limit.
What you can do after
Operate/configure it to bound a DAG’s concurrent tasks while checking whether run limits, pools, or waiting sensors need separate attention.
How it connects
Connects DAG concurrency to backfilling, max_active_runs, sensors, pools, and global parallelism.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.