Snowflake is a cloud data platform for storing, processing, developing applications with, and analyzing data. It supports data warehousing, data lakes, data marts, analytics, and data science workloads, making it a target platform for pipelines that ingest source data, transform it, and deliver it to downstream consumers. section 1.1 section 1.5
Its architecture separates storage from compute. Snowflake manages compressed columnar data in cloud storage and exposes database objects through SQL. Virtual warehouses provide isolated, configurable compute for individual workloads, while cloud services handle authentication, access control, infrastructure management, query optimization, metadata, and security. This separation lets multiple pipelines, query engines, analytics workloads, and data science solutions access stored data without sharing the same compute resources. section 1.1.1
Figure 1.1 The three layers of the Snowflake architecture are database storage, query processing, and cloud services.
The three-layer view distinguishes database storage, query processing through virtual warehouses, and cloud services that coordinate platform operations. section 1.1.1
Snowflake provides pipeline-oriented capabilities for moving and preparing data:
- Load structured and semistructured files from AWS S3, Azure Blob Storage, or GCP buckets, including CSV, JSON, XML, and Parquet; it also supports unstructured files and file metadata. section 1.1.2
- Use Snowpipe for continuous ingestion when cloud-provider notifications signal that new files are available. section 1.1.2
- Schedule SQL, stored procedures, or Snowflake Scripting with tasks. Task dependencies can form larger processing pipelines. section 1.1.2
- Build native pipelines with Snowpark APIs for Python, Scala, or Java, including open-source libraries where needed. section 1.1.2
- Restrict sensitive data with masking policies and row-level access policies, including multitenant data scenarios. section 1.1.2
A typical implementation stages an input file, loads it into a staging table with COPY, merges rows into a target table, transforms the target data with SQL, and schedules daily execution with a task. Downstream delivery also includes a data model suited to the consuming tool, discoverable metadata such as table and column comments, and operational metrics such as refresh time, record counts, and data-quality measures. section 2.0 section 1.2.3
Scheduled CSV order pipeline
A bakery collects daily emailed orders in a local CSV file. The pipeline stages the file in Snowflake, loads raw rows to a staging table, merges them into a target table, transforms them into a reporting-ready model, and runs on a daily schedule. The pattern establishes a simple baseline before adding security, governance, software engineering, or DataOps components. section 2.0
Operational choices affect resilience, performance, and cost. Zero-copy cloning creates environments without additional storage costs, while time travel enables access to prior data states within a configurable period. For demanding queries, clustering, search optimization, materialized views, or dynamic tables can improve performance. Costs are consumption-based: compute uses credits, while stored objects, staged files, optimization paths, and time-travel history consume storage. section 1.1.1 section 1.2.3 section 9.1.1
