An event notification service delivers metadata messages when specified events occur in cloud storage or other cloud resources. For continuous ingestion, it notifies Snowpipe when new files arrive; the files remain in cloud storage until ingestion. section 5.3 section 3.2
A notification-driven pipeline uses an external stage and storage integration to expose cloud files, then uses a queue to alert Snowpipe. Snowpipe consumes the notification and loads the new files into a staging table, where downstream processing can materialize the data. section 5.0 section 5.3
Key operating characteristics:
- Carries file metadata, such as the names of newly arrived files, rather than the file contents. section 5.3
- Stores arrival events in a queue that Snowpipe uses to identify files for loading. section 5.3
- Can automatically refresh external-stage directory-table metadata when files are added or removed. section 3.2
- Supports cloud messaging as an alternative to notifying Snowpipe through its REST endpoints. section 5.3
Figure 5.1 Data pipeline that continuously ingests data from JSON files in external cloud storage with Snowpipe. A storage integration object exposes the files in an external stage. A notification integration object configured with cloud messaging notifies Snowpipe when new files arrive so they can be ingested into the staging table with minimum delay. The JSON data in the staging table is flattened and materialized by a dynamic table.
Trace the complete flow from external JSON files and storage integration through notification-driven Snowpipe ingestion into a dynamic table. section 5.0
