This chapter covers:
- Designing a flexible and scalable 6-layer data platform architecture that supports both batch and streaming data, meets the needs of different data consumers, and has the right foundational components for easier ongoing management.
- Choosing the best tools and services to implement a modern cloud data platform in any of AWS, GCP, and Azure.
Chapter 2 covered setting up a simple data platform made up of a data lake and data warehouse in the cloud, with simple batch pipelines to ingest data. It also laid out the pros and cons of a data lake vs a data warehouse versus a combination of the two to produce the best analysis outcomes.
In this chapter, we’ll build upon the data platform architecture concepts introduced in Chapters 1 & 2 and we’ll layer on top of it some of the critical and more advanced functionality needed for most data platforms today. Without this added layer of sophistication your data platform would work but it wouldn’t scale easily, nor would it meet the growing data velocity challenges discussed in Chapter 1. It would also be limited in terms of the types of data consumers (people and systems who consume the data from the platform) it supports, as they too are growing in both numbers and variety.
We will take a deeper dive into a more complex cloud data platforms architecture: