This key chapter puts the “engineering” in data engineering. DevOps practices allow us to build reliable, reproducible systems. One of the principles you will see repeated throughout the book is tracking everything in source control and deploying everything automatically. Figure 3.1 highlights the layer we will cover in this chapter.
Figure 3.1 Tracking everything in source control and automatically deploying is foundational to a robust system.
In this chapter, we will talk about DevOps and how it became an industry standard for software engineering. We’ll see what learning we can take from that and apply it to the world of data and data platforms. We’ll explore Azure DevOps, the Azure offering in the DevOps space, which provides an integrated, one-stop-shop service for all our needs.
First, we will apply DevOps to infrastructure and see how we can deploy Azure Data Explorer (ADX) automatically from source control, including all the configuration we went through in the previous chapter. Next, we will apply DevOps to analytics and see how we can deploy tables and queries from source control. Let’s start with a discussion on DevOps: what it is and why it matters.