2 Creating Your First Data Pipeline

In this chapter, you will learn how to build your first data pipeline in Snowflake. We will ingest data from a CSV file into Snowflake, transform the data from its raw form into a target data model required for reporting, and automate the pipeline. This chapter does not include underlying data pipeline components such as security, data governance, software engineering, or DataOps introduced in Chapter 1 because we want to keep it simple initially. These topics are described in more detail in subsequent chapters. To get started with data engineering, we will create a simple data pipeline that will illustrate the core Snowflake functionality required to ingest data from a file into Snowflake and transform the data.

2.1 Setting up your Snowflake Account

2.2 Staging a CSV File

2.3 Loading Data from a Staged File into a Target Table

2.3.1 Loading Data from a Staged file into a Staging Table

2.3.2 Merging Data from the Staging Table into the Target Table

2.4 Transforming Data with SQL commands

2.5 Automating the Process with Tasks

2.6 Summary