liveBook · Manning

REFRESH command

As explained in Snowflake Data Engineering

The REFRESH command instructs Snowpipe to detect eligible files that were already present in cloud storage.

REFRESH updates an existing Snowpipe so it detects eligible files that were already in cloud storage before the pipe was created. It complements event-driven ingestion, which otherwise responds only to newly arriving files. section 5.3.3

Use REFRESH with these operational boundaries:

  • Run `ALTER PIPE SPEEDY_PIPE REFRESH;` after creating the pipe to discover previously uploaded files. section 5.3.3
  • Detection is limited to files uploaded within the previous seven days; load older files with a bulk `COPY` command. section 5.3.3
  • Allow a few minutes for ingestion, then query the target staging table to verify loaded rows. section 5.3.3
  • Check execution state, pending files, and the last ingestion through `SYSTEM$PIPE_STATUS`; inspect table load history with `COPY_HISTORY`. section 5.3.3 section 5.5
alter pipe SPEEDY_PIPE refresh;

Run the pipe alteration to refresh discovery of files already present in the configured cloud-storage location. section 5.3.3

How Snowflake Data Engineering covers it

How deep it goes
Defines the command, seven-day limit, verification delay, and monitoring functions, but defers older-file loading to bulk COPY.
Examples
Uses SPEEDY_PIPE to refresh three earlier JSON order files into SPEEDY_ORDERS_RAW_STG, then verifies three resulting rows.
What it assumes
Assumes an existing pipe, configured storage stage, and files in cloud storage; teaches the separate bulk-load path for older files.
What you can do after
Operate Snowpipe to backfill recent pre-existing files and diagnose ingestion through status and load-history queries.
How it connects
Connects Snowpipe with COPY commands, cloud storage, event notifications, bulk loading, and pipe-status monitoring.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Snowflake Data Engineering. The section links point to the parts of the book it draws on.