Dagster orchestrates ingestion and transformation code into modular data pipelines. Its central abstraction is the software-defined asset: a table, file, or model described in Python, including how to create or update it and which assets it depends on. Assets can then be scheduled, monitored, and traced through lineage and provenance. section 8.4
A practical Dagster workflow connects:
- External data sources to DuckDB tables through Python asset functions. section 8.4
- Dependent assets to explicit execution order; a derived asset waits until its input asset succeeds. section 8.4.3
- DuckDB SQL with Python and pandas computations for ingestion, cleaning, augmentation, and derived values. section 8.4.4
Materializing a tennis-data pipeline
Define an asset that reads ATP player data into DuckDB, then define a dependent asset that adds a name_full column by concatenating name_first and name_last. Launch the development UI with `dagster dev -m atp`, materialize the assets, or run `dagster job execute -m atp --job atp_job`. Dagster starts independent assets together and runs the player-name transformation after its dependency completes. Querying DuckDB then verifies the loaded rows and derived column. section 8.4.2 section 8.4.3
dagster dev -m atp
Use the Dagster development command to inspect the job, schedule, and asset graph in the local UI before materializing the pipeline. section 8.4.2
