liveBook · Manning

Target lag

As explained in Snowflake Data Engineering

Target lag is a dynamic-table setting that specifies the maximum permitted delay between the dynamic table and its source tables.

Target lag is the dynamic-table setting that defines the maximum permitted delay between a dynamic table and the tables queried by its definition. Set it with TARGET_LAG when creating the table, alongside the virtual warehouse used for refreshes. Snowflake maintains the table in the background, so the transformed query result is materialized without separately scheduling population tasks. section 5.4 section 5.5

A value of '1 minute' means the dynamic table should lag its source data by no more than one minute. After creation, it initially contains the query result; when data in queried tables changes, the dynamic table updates accordingly. The setting applies to tables that flatten staged data, normalize joined staging data, or publish summarized reporting data. section 5.4 section 12.4 section 12.4.2

create dynamic table SPEEDY_ORDERS
  target_lag = '1 minute'
  warehouse = BAKERY_WH
  as 
  select     #1
  order_id,
  order_datetime,
  value:"Item"::varchar as baked_good_type,
  value:"Quantity"::number as quantity,
  source_file_name,
  load_ts
from SPEEDY_ORDERS_RAW_STG,
lateral flatten (input => items);

This definition combines a one-minute target lag with BAKERY_WH and a query that flattens order items from a staging table. section 5.4

Use target lag as part of each dynamic table definition:

  • Flatten staged JSON order data into SPEEDY_ORDERS with TARGET_LAG = '1 minute', then query the table after new files arrive. section 5.4
  • Join staged orders to product and partner data when creating ORDERS_TBL; the one-minute lag applies when any queried table changes. section 12.4
  • Create a reporting summary that filters product history to current values, then materialize it as ORDERS_SUMMARY_TBL with a one-minute lag. section 12.4.2
select *
from table(information_schema.dynamic_table_refresh_history())
order by refresh_start_time desc;

Inspect dynamic-table refresh history to view refresh activity after configuring the table. section 5.4

How Snowflake Data Engineering covers it

How deep it goes
Defines the parameter and demonstrates one-minute SQL configurations, but does not detail refresh scheduling internals.
Examples
Uses flattened order items, normalized orders joined to product and partner data, and a current-value order summary.
What it assumes
Uses SQL SELECT queries, joins, and CREATE DYNAMIC TABLE statements to define materialized results.
What you can do after
Operate/configure dynamic tables with a target lag and refresh warehouse for transformed query results.
How it connects
Connects dynamic tables with virtual warehouses, declarative data transformation, staging tables, and reporting summaries.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Snowflake Data Engineering. The section links point to the parts of the book it draws on.