liveBook · Manning

Asset-based schedule

As explained in Data Pipelines with Apache Airflow, Second Edition

Asset-aware scheduling is a workflow scheduling technique that triggers work based on the availability or update state of data assets rather than only on fixed times.

Suppose one team ingests an events data set and several other teams process it. Giving every consumer a fixed schedule can waste runs when the data has not changed—or start processing before ingestion finishes. Having the producer trigger each consumer directly avoids the timing problem but leaves it maintaining a growing list of downstream workflows. An asset-based schedule connects them through updates to the data instead. section 4.1

In Airflow, an asset is a virtual reference identified by a URI or string ID, such as `file:///data/events` or `s3://example_bucket/example.csv`. A producer DAG declares the asset as an output and emits an asset event when it updates the data. A consumer DAG schedules itself on that asset; Airflow starts it when the update occurs and makes the triggering event available to the consumer. The identifier names the dependency: Airflow does not infer the asset’s contents or location from it. section 4.2

Wait for two data sets

For a consumer combining two event feeds, define `events_1 = Asset("file:///data/events_04_1")` and `events_2 = Asset("file:///data/events_04_2")`, then set `schedule=[events_1, events_2]` on its DAG. By default, the consumer waits until *both* assets have received an event; their update order does not matter. Use `schedule=(asset_1 | (asset_2 & asset_3))` when either one asset or a pair of others should trigger a run. section 4.7

Figure 4.11 Every star icon indicates an asset update for either the first or second data set. Only when both data sets have been updated is a DAG run for our consumer DAG triggered. The order in which the assets are updated doesn’t matter.

The update timeline shows why an event on just one of two required assets does not start the consumer. section 4.7

Asset events can carry extra information for downstream DAGs, which can inspect `triggering_asset_events` to select inputs. If a producer run leaves its asset unchanged, it can raise `AirflowSkipException` to avoid triggering consumers. Do not assume that events from multiple assets describe the same execution date: Airflow does not match upstream scheduling dates when combining them. Check and handle mismatches in the consumer when that alignment matters. A DAG can also combine asset- and time-based scheduling when it needs both update-driven and periodic runs. section 4.9 section 4.7

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Works through asset identifiers, producer-consumer events, multi-asset conditions, and date-alignment caveats; does not describe scheduler internals.
Examples
Uses an events data set split between producer and consumer DAGs, then combines two feeds with an AND or OR schedule.
What it assumes
Assumes familiarity with DAGs and time-based schedules, while defining assets and their producer-consumer relationship.
What you can do after
Apply it via Airflow assets to schedule consumer DAGs on updates and choose conditions for multiple inputs.
How it connects
Connects DAGs, data assets, asset events, time-based scheduling, and execution dates.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.