3 DevOps
In this chapter:
- Bringing DevOps to data engineering
- Introducing Azure DevOps
- Deploying infrastructure
- Deploying analytics
This is a key chapter which puts the “engineering” in data engineering. DevOps practices allow us to build reliable, reproducible systems. One of the principles you will see repeated throughout the book is tracking everything in source control and deploying everything automatically.
Figure 3.1 highlights the layer we are going to cover in this chapter.
Figure 3.1 Tracking everything in source control and deploying automatically is foundational to a robust system.
We will talk about DevOps and how it became an industry standard for software engineering. We’ll see what learnings we can take from there and apply to the world of data and data platforms.
We’ll explore Azure DevOps, the Azure offering in the space, which provides an integrated, one-stop-shop, service for all our needs.
First, we will apply DevOps to infrastructure and see how we can deploy Azure Data Explorer automatically from source control, including all the configuration we went through in the previous chapter. Next, we will apply DevOps to analytics and see how we can deploy tables and queries from source control.
Let’s start with a discussion on DevOps – what it is and why it matters.