A source system is an operational or external system from which data is extracted for downstream storage, transformation, and consumption. Data engineers assess its business meaning, access method, format, security requirements, and effect on pipeline design. section 1.2 section 1.2.1
Identify the source before implementing ingestion:
- Classify the source as relational database tables, cloud-storage files, application API output, IoT streams, secure data sharing, or Snowflake Marketplace data. section 1.2.1
- Determine whether multiple sources must be unified into a single model for downstream consumers. section 1.2.1
- Prefer incremental extraction of new and changed records using timestamps, database change tracking, or another agreed change-detection method. section 1.2.1
- Use Snowflake streams when the source does not provide an easy way to identify inserts, updates, or deletes. section 1.2.1
- Choose commercial or open-source extraction tools, or custom code, with solution architects and other stakeholders. section 1.2.1
Operational sources for a bakery pipeline
A bakery pipeline can simulate relational tables from its information system, including business partners and baked goods. Those tables join file-based CSV or JSON inputs and may be combined with other sources before ingestion into Snowflake. section 11.0
Figure 1.2 Data engineering components with Snowflake include extracting data from source systems, ingesting the data into Snowflake, performing transformations, and presenting the data to downstream consumers, incorporating underlying components and best practices related to security, data modeling, data governance, orchestration, software engineering, and DataOps.
Use the pipeline overview to trace how source extraction connects to ingestion, transformation, presentation, security, governance, orchestration, and DataOps. section 1.2
