Chapter 2 covered setting up a simple data platform made up of a data lake and a data warehouse in the cloud, with simple batch pipelines to ingest data. It also laid out the pros and cons of a data lake versus a data warehouse versus a combination of the two to produce the best analysis outcomes.
In this chapter, we’ll build on the data platform architecture concepts introduced in chapters 1 and 2, and we’ll layer on top of those some of the critical and more advanced functionality needed for most data platforms today. Without this added layer of sophistication, your data platform would work, but it wouldn’t scale easily, nor would it meet the growing data velocity challenges discussed in chapter 1. It would also be limited in terms of the types of data consumers (people and systems who consume the data from the platform) it supports, as they too are growing in both numbers and variety.