A directed acyclic graph (DAG) organizes dependent work so that flow moves in one direction and never returns to an earlier step. In Snowflake, a task graph is a DAG: one scheduled root task starts a pipeline, while dependent tasks run only after all predecessor tasks complete successfully. A task may have multiple parents and multiple children. section 13.3 section 13.6
Use a DAG to orchestrate a data pipeline whose ingestion, transformation, logging, and notification steps have ordering requirements. The root task can perform setup or start logging; a finalizer task performs end-of-pipeline actions and runs even when preceding tasks fail. Dependent tasks should not retain independent schedules: they run through their AFTER dependencies. section 13.3 section 13.3.3
Design a task DAG around explicit operational dependencies:
- Identify externally controlled work, such as source-system updates, separately from steps the pipeline must automate. section 13.3.1
- Create independent tasks for work that can proceed in parallel, then place both after their shared predecessor and before the finalizer. section 13.3.1
- Schedule only the root task; connect downstream tasks with dependencies so successful predecessor completion controls execution. section 13.3
- Suspend the root task when changing the graph or preventing unneeded scheduled runs; dependent tasks cannot run without their predecessor. section 13.3.4
Incremental partner and product pipeline
A root task starts an incremental pipeline. Separate tasks load PARTNER_STREAM into PARTNER_TBL and PRODUCT_STREAM into PRODUCT_TBL; because they are independent, they can run in parallel after the root. A finalizer records completion and sends the end notification. Streams track changes automatically, while the task graph automates the warehouse-layer loads. section 13.3.1 section 13.3.3
Inspect a graph in Snowsight to understand its scope and dependencies. For execution problems, check TASK_HISTORY() for failed, skipped, or predecessor states; also verify that tasks are resumed, dependencies are correct, schedules have reached their run time, and stream-driven tasks have data to process. section 13.3.4 section 13.5
Table 13.6 Steps performed when ingesting and transforming the partners and products data incrementally
|
Step number
|
Step description
|
Consideration for orchestration design
|
|---|---|---|
|
1
|
The data integration tool updates the
PARTNER and
PRODUCT tables in the staging layer.
|
This is done externally by the data integration tool, and we have no control over it in the data pipeline (we simulate this step by inserting or updating data manually).
|
|
2
|
Two streams named
PARTNER_STREAM and
PRODUCT_STREAM keep track of new data since the last pipeline execution.
|
The streams detect and track changes automatically. We don’t have to do anything specific in the data pipeline.
|
|
3
|
Inserts data from the
PARTNER_STREAM stream into the
PARTNER_TBL table in the warehouse layer
|
This step must be automated with a task.
|
|
4
|
Inserts data from the
PRODUCT_STREAM stream into the
PRODUCT_TBL table in the warehouse layer
|
This step must be automated with a task.
|
Review the incremental ingestion steps to distinguish external source updates and automatic stream tracking from the two task-driven loads that can run in parallel. section 13.3.1
