liveBook · Manning

Snowflake task

As explained in Snowflake Data Engineering

A Snowflake task is a scheduled Snowflake object that executes SQL statements, stored procedures, or Snowflake Scripting blocks.

A Snowflake task is a user-defined object that runs a SQL statement, stored procedure, or Snowflake Scripting block. It can start at a specified time, recur at an interval, or run after a predecessor finishes. Tasks provide native scheduling and execution for data-pipeline orchestration. section 13.1

Use a task to automate pipeline steps that would otherwise be run manually, such as loading staged files, merging data, and producing reporting summaries. A task needs compute: either a chosen virtual warehouse or Snowflake-managed serverless compute. section 2.5 section 13.1

Scheduled order-processing task

A PROCESS_ORDERS task runs every 10 minutes during testing. Its Snowflake Scripting block truncates a staging table, loads files with COPY INTO, merges orders into a target table, and refreshes a summary table. After validation, suspend it, change the schedule to 11 p.m. UTC, then resume it; new tasks begin suspended. section 2.5

use database BAKERY_DB;
use schema ORDERS; 
create task PROCESS_ORDERS
  warehouse = BAKERY_WH
  schedule = '10 M'
as
begin
  truncate table ORDERS_STG;
  copy into ORDERS_STG...         #1
  merge into CUSTOMER_ORDERS...       #2
  truncate table SUMMARY_ORDERS;
  insert into SUMMARY_ORDERS...     #3
end;

This task definition shows a warehouse-backed schedule and a single scripting block that groups sequential pipeline SQL operations. section 2.5

For production task orchestration:

  • Store tasks and related logging, error-handling, and configuration objects in a dedicated orchestration schema, separate from pipeline data-layer schemas. section 13.1.1
  • Model dependent steps as a task graph (DAG): schedule the root task, and let child tasks run after predecessors complete successfully. Streams can supply incremental-processing conditions. section 13.1 section 13.6
  • Grant EXECUTE TASK to the role that creates and runs tasks, then test tasks manually before relying on a schedule. section 13.1.2 section 13.6
  • Inspect TASK_HISTORY() for execution state, timing, errors, and the next scheduled run. For failures, validate task-body SQL, status, dependencies, predecessors, schedule, and available stream data. section 13.5 section 2.5
  • Use logging and a finalizer task to summarize completed task work and send pipeline-completion email notifications when needed. section 13.4.2

Scheduled tasks do not overlap: if a prior scheduled run is still executing when the next time arrives, the later run is skipped. A stream-conditioned task can also be skipped when no data is available. section 13.6

How Snowflake Data Engineering covers it

How deep it goes
Builds from a single scheduled SQL block to dependency graphs, streams, monitoring, troubleshooting, logging, and finalizer notifications.
Examples
Uses bakery order CSV processing and a multi-task JSON ingestion pipeline with logs and completion email.
What it assumes
Assumes SQL pipeline steps and role-based access control while explicitly granting task-execution privileges and configuring task compute.
What you can do after
Operate/configure it by creating, scheduling, resuming, testing, monitoring, and troubleshooting single tasks and dependent task graphs.
How it connects
Connects tasks with Snowflake Scripting, stored procedures, virtual warehouses, serverless compute, streams, task graphs, and TASK_HISTORY().

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Snowflake Data Engineering. The section links point to the parts of the book it draws on.