In this chapter, we’ll look at the final pieces of core infrastructure for our data platform: orchestration and monitoring. DevOps is where we store all our code and configurations and from which we deploy our services. The storage layer is where we ingest data and on top of which we run our workloads. The orchestration layer handles data movement and all other automated processing. Figure 4.1 highlights the platform layer we’ll focus on in this chapter.
Figure 4.1 The orchestration layer handles scheduling for all tasks and data movement into and out of the data platform.
We’ll start with a real-world scenario: ingesting the Bing COVID-19 Open Research Dataset into our data platform. Microsoft provides several open datasets for everyone’s use. One of these tracks COVID-19 cases. We’ll use Azure Data Factory (ADF) to create a pipeline to bring this dataset into an Azure Data Explorer (ADX) cluster.