A directed acyclic graph (DAG) models a workflow as one-way task dependencies without loops. Each task waits for its prerequisites, making execution order predictable, auditable, and easier to debug. section 11.1
In lakehouse maintenance, a DAG can:
- Check Iceberg metadata and trigger compaction only when small-file or snapshot thresholds are exceeded. section 11.1
- Enforce sequences such as compaction before snapshot expiration and orphan-file removal. section 11.1.1
- Add retries, alerts, conditional logic, per-table parameters, and observability metrics. section 11.1.1 section 11.1.5
Metadata-aware compaction
An Airflow DAG queries an Iceberg files table, counts small files, and runs Spark's rewrite_data_files procedure only when the count exceeds 500. A separate dbt workflow can expire snapshots after transformation, while Python can publish snapshot counts to Prometheus. section 11.1.5
Figure 11.1 A directed acyclic graph (DAG) illustrates a non-looping order of tasks. In this example, tasks 2 and 3 can’t begin until task 1 is complete, and task 4 can’t begin until tasks 2 and 3 are complete. This ensures that a task doesn’t begin until all other dependent tasks are complete.
Shows a branching dependency pattern in which tasks 2 and 3 wait for task 1, and task 4 waits for both branches. section 11.1
