chapter six

6 Triggering workflows

 

This chapter covers

  • Waiting for certain conditions to be met with sensors
  • Deciding how to set dependencies between tasks in different DAGs
  • Executing workflows via the CLI and REST API
  • Triggering DAGs when data updates using the Dataset

In chapter 3, we explored how to schedule workflows in Airflow based on a time interval. The time intervals can be given as convenience strings (e.g., "@daily"), timedelta objects (e.g., timedelta(days=3) to trigger a workflow every 3 days), or cron strings (e.g., "30 14 * * *"). These are all notations to instruct the workflow to trigger at a certain time or interval. Airflow will compute the next time to run the workflow given the interval and start the first task(s) in the workflow at the next date and time.

6.1 Polling conditions with sensors

6.1.1 Polling custom conditions

6.1.2 Sensors outside the happy flow

6.2 Triggering other DAGs

6.2.1 Backfilling with the TriggerDagRunOperator

6.2.2 Polling the state of other DAGs

6.2.3 Trigger a DAG on data updates

6.3 Starting workflows with REST/CLI

6.4 Summary