A data pipeline layer is a logical stage through which data moves during extraction, staging, warehousing, transformation, and presentation. Each layer produces outputs that can be stored as Snowflake objects such as tables, views, materialized views, or dynamic tables. section 11.1.4 section 11.0
Organize layers physically according to source separation, access boundaries, and consumer needs:
- Represent every layer as a schema within one database. section 11.1.4
- Place each layer in a separate database when stronger separation is useful. section 11.1.4
- Use separate extraction databases for source systems, combine data in the warehouse layer, and expose presentation data by business unit. section 11.1.4
- Treat security as a design constraint by limiting consumers to authorized layer objects. section 11.1.4
Single-database implementation
A Snowflake implementation can place the transformation layers in BAKERY_DB, using EXT, STG, DWH, and MGMT schemas for extraction, staging, warehouse, and management or presentation functions. section 11.1.4 section 11.0
Figure 11.4 The data transformation layers used in the examples in this chapter are implemented as a single database named BAKERY_DB, with each layer represented as a schema named EXT, STG, DWH, and MGMT.
Use the layer diagram to inspect how one Snowflake database maps logical pipeline stages to schemas. section 11.1.4
