12 Integrating Data Factory with SQL Database

In the last chapter, you learned about creating and configuring Azure SQL Database (SQLDB). Relational databases like SQLDB provide an accessible endpoint for queries into both the Serving layer and Speed layer of a Lambda architecture analytics system. In chapter 6, you saw the real-time part of the workflow, with Stream Analytics calculations that flowed into SQLDB. Chapter 10 demonstrated using Azure Data Factory (ADF) to automate the batch-processing part using Azure Data Lake Analytics (ADLA). SQLDB sits at the center of this web of data flows and forms the primary service of the Serving layer. You can see the entire design in figure 12.1.

In this chapter, you’ll learn how to move data in Azure services into SQLDB. This includes the outputs of ADLA batch processing, data in Blob Storage, and data in other SQLDBs. Providing data via an RDBMS gives more users access, with a wider range of tools, than data files only.

12.1 Before you begin

12.2 Importing data with external data sources

12.2.1 Creating a database scoped credential

12.2.2 Creating an external data source

12.2.3 Creating an external table

12.2.4 Importing Blob files

12.3 Importing file data with ADF

12.3.1 Authenticating between ADF and SQLDB

12.3.2 Creating SQL Database linkedservice

12.3.3 Creating datasets

12.3.4 Creating a copy activity and pipeline