6 Triggering workflows with external input

 

This chapter covers

  • Waiting for certain external conditions to be met with sensors
  • Deciding how to set dependencies between tasks in different DAGs
  • Executing workflows via the CLI and REST API
  • Triggering DAGs when messages are published on a Message Queue

In chapter 3, we explored how to schedule workflows in Airflow based on a time interval. The schedule can be defined with fixed triggers (e.g. CronTriggerTimetable) or intervals (e.g. CronDataIntervalTimetable). The different types of triggers are all intended to set the workflow to trigger at a certain time or interval. Airflow will compute the next time to run the workflow given the interval and start the first task(s) in the workflow at the next date and time.

In chapter 4, we explored other mechanisms for triggering workflows. The focus here was on event-driven scheduling, where no fixed interval or time was defined but a specific event occurred which caused an Airflow workflow to run.

6.1 Polling conditions with sensors

6.1.1 Polling custom conditions

6.1.2 Sensors outside the happy flow

6.2 Starting workflows with REST/CLI

6.3 Triggering workflows with messages

6.4 Summary