liveBook · Manning

Asset

As explained in Data Pipelines with Apache Airflow, Second Edition

An asset is a virtual reference identified by a string ID or URI that represents a data object or other consumable or producible resource.

An asset is a virtual reference to something a workflow produces or consumes, such as a data set, table, or machine learning model. Its URI acts as a string ID: Airflow does not infer the resource’s location or contents from it. Choose an identifier that uniquely identifies the resource for its consumers. section 4.2

Assets make dependencies between DAGs explicit. A producer updates an asset and emits an asset event; a consumer scheduled on that asset runs when it is updated. This separates a time-scheduled data-fetching DAG from downstream processing without giving each consumer its own polling schedule. Multiple consumers can reference the same asset. section 4.0 section 4.2

Produce and consume an events data set

Use the same asset ID in both DAGs: from airflow.sdk import Asset events = Asset("file:///data/events_01") # On the producer's task that updates the data set: fetch_events = PythonOperator(..., outlets=[events]) # On the consumer DAG: with DAG(dag_id="events_consumer", schedule=[events], ...): ... The producer can retain its daily schedule. Its updating task emits an asset event, which triggers the consumer. Inspect the asset and its events in the Airflow UI’s Assets tab. section 4.3 section 4.4

An asset-triggered run does not provide the date or interval parameters of a time-scheduled run. If processing needs to select a dated input file, use `triggering_asset_events` to inspect the event and its producing DAG run rather than templating the consumer’s `logical_date`. section 4.4

A consumer can also schedule on two assets and combine their input files. In that arrangement, it runs after both assets have received updates, regardless of update order. Select each asset’s event explicitly when constructing its input path; conditional asset expressions can express other dependencies. section 4.7

Figure 4.11 Every star icon indicates an asset update for either the first or second data set. Only when both data sets have been updated is a DAG run for our consumer DAG triggered. The order in which the assets are updated doesn’t matter.

The update timeline makes the two-asset trigger condition visible: an update to just one input does not start the consumer. section 4.7

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Defines the identifier and works through producer outputs, consumer schedules, triggering events, and multiple-input conditions; does not describe asset storage internals.
Examples
Uses an events data set, PythonOperator outlets, an asset-scheduled statistics DAG, and a two-source data combination.
What it assumes
Builds on DAGs, scheduled runs, and PythonOperator tasks while defining assets from first principles.
What you can do after
Apply it via Airflow to trigger consumer DAGs from produced data and select inputs using triggering asset events.
How it connects
Connects DAG scheduling, producer–consumer workflows, asset events, and Jinja templating.

Also explained in

Compare how 2 books explain asset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.