1 Data Engineering with Snowflake
This chapter covers
- Featuring Snowflake for data engineering
- Examining the responsibilities of a Snowflake data engineer
- Constructing data pipelines with Snowflake
- Data engineering with Snowflake applications
Organizations in just about every industry collect data, often in massive amounts. This data can be used by data analysts, data scientists, and other business professionals to extract insights that aid in decision making. Raw data is rarely suitable for consumption directly from the source. For most practical purposes, data must be transformed according to business rules, reshaped to fit a particular business need, and optionally enriched with external data before it is ready for downstream consumers.
Data engineering is the practice of building solutions that extract data from source systems, transform the data into useful information, and present the harmonized data to users for downstream consumption. The abbreviations ETL (extract – transform – load) or ELT (extract – load – transform) are also commonly used to denote these solutions.
Data engineers build pipelines that make data available to data analysts, data scientists, and other consumers who rely on this data to get their job done. Ensuring that high-quality data is delivered when users need it is crucial for successful analytics. Therefore, data engineers are an indispensable link in the data analytics realm.