liveBook · Manning

Trigger rule

As explained in Data Pipelines with Apache Airflow, Second Edition

A trigger rule is a condition that determines whether an Airflow task is ready to execute based on the execution status of its dependencies.

A trigger rule determines when an Airflow task is ready to run from the states of its upstream tasks. The default, `all_success`, requires every parent to succeed. Airflow checks these conditions during a DAG run and schedules a task when it becomes ready. section 6.4.1 section 6.4

Consider a sales pipeline that chooses between two ERP systems, then joins the selected sales data with weather data. A `BranchPythonOperator` skips the unchosen sales path. If the join depends on both paths, its default `all_success` rule cannot be met, so Airflow skips the join and its downstream tasks. section 6.2.2

Let the branches rejoin

Set `trigger_rule="none_failed"` on the join task. It waits for every parent to finish, accepts successful or skipped parents, and still blocks the join if a parent fails. For a clearer DAG, put an `EmptyOperator(task_id="join_erp_branch", trigger_rule="none_failed")` after the two sales-cleaning paths, then connect it to the dataset join. The changed rule stays on the branch join rather than on the later data-processing task. section 6.2.2 section 6.4.3

Choose the rule for the result you need:

  • Keep `all_success` when downstream work requires every parent to succeed. A failed parent can leave dependent tasks in `upstream_failed`; a skipped parent can propagate skips. section 6.4.1 section 6.4.2
  • Use `all_done` for cleanup that must run after every parent finishes, regardless of result. `one_failed` and `one_success` can react as soon as one parent meets the condition, without waiting for all parents. section 6.4.3
  • For conditional deployment, a condition task can raise `AirflowSkipException`; the downstream deployment task’s default `all_success` rule then prevents deployment for that run. section 6.3.2
Listing 6.14 Fixing the trigger rule of join_datasets
join_datasets = PythonOperator(
    ...,
    trigger_rule="none_failed",
)   

The operator listing shows where to set `trigger_rule="none_failed"` when a task directly joins branching paths. section 6.2.2

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Explains scheduler checks, state propagation, and several rule choices through DAG examples; defers a deeper survey of trigger rules.
Examples
Shows an ERP branch rejoining a sales-and-weather pipeline, a latest-only deployment condition, and cleanup after upstream tasks finish.
What it assumes
Assumes basic familiarity with Airflow DAGs, tasks, and dependencies; defines trigger rules from upstream task states.
What you can do after
Operate/configure Airflow joins and conditional tasks with appropriate trigger rules while avoiding unintended downstream skips.
How it connects
Links trigger rules to DAG dependencies, task states, BranchPythonOperator, AirflowSkipException, and conditional deployment.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.