A data transformation layer organizes intermediate pipeline outputs between source ingestion and clean data for downstream consumers. Outputs may be stored physically or transformed virtually. section 11.1.3
Common layers follow this sequence, although pipelines may omit or add layers:
- Extraction layer: hosts Snowflake stages, external tables, or raw structured and semistructured source data. section 11.1.3
- Staging layer: consolidates data and applies light transformations such as filtering, ingestion timestamps, surrogate keys, and changed-data identification, without business logic. section 11.1.3
- Data warehouse layer: provides a common area where data from multiple sources is collected and modeled consistently. section 11.1.3
- Presentation layer: delivers clean transformed data for downstream consumers. section 11.1.3
Pipeline design choice
A pipeline can build presentation-layer data marts without a data warehouse, or omit the extraction layer when an integration tool performs extraction and transformation through ETL. section 11.1.3
Figure 11.3 The most common data transformation layers in data pipelines are the extraction layer, the staging layer, the data warehouse layer, and the presentation layer.
Use the figure to compare the extraction, staging, data warehouse, and presentation layers as a connected pipeline. section 11.1.3
