Data ingestion imports source data into Snowflake so it can be stored, queried, and transformed. Files can come from AWS S3, Azure Blob Storage, or Google Cloud Storage and may use structured or semistructured formats such as CSV, JSON, XML, and Parquet. Unstructured files can also be accessed through file URLs and metadata. section 1.1.2
A common cloud-file pipeline exposes a storage location through an external stage, loads files into a staging table, then inserts or merges staged rows into a target table. Authenticate to the provider with a storage integration where appropriate, or provide credentials; provider-specific stage syntax differs. section 3.0
Figure 3.1 A data pipeline that takes CSV files from a cloud storage location makes the files available to Snowflake in an external stage, either by using a storage integration object or by providing credentials, loads the data into a staging table, and inserts or merges the data from the staging table into the target table.
Follow the baseline flow from cloud storage through an external stage and staging table to an insert or merge into the target table. section 3.0
Choose an ingestion approach according to arrival patterns and query needs:
- Use COPY on an external stage for bulk loading from cloud storage into Snowflake tables. section 3.0
- Use external tables to query files in an external stage; add materialized views when query performance needs improvement. section 3.0
- Use Snowpipe when files must be loaded as they arrive. Cloud-provider notifications tell Snowpipe about new files, reducing ingestion delay. section 1.1.2 section 5.0
- Use scheduled tasks when SQL statements, stored procedures, or scripting blocks should run on a recurring schedule or depend on other tasks. section 1.1.2
For high-volume loads, ingestion performance and cost depend on how files are sized and organized in cloud storage. Small examples may hide these concerns, but pipelines receiving many files and large volumes need deliberate file preparation. section 3.3
Continuous JSON order feed
A delivery service places JSON files in external cloud storage. A storage integration makes the files available through an external stage, and cloud messaging notifies Snowpipe when each file arrives. Snowpipe loads a staging table with minimum delay; the JSON can then be flattened and materialized in a dynamic table. Check the staging table after uploading a file, allowing time for continuous loading. Preserve the expected JSON structure and use unique file names when repeating uploads. section 5.0 section 5.3.4
