concept data platform architecture in category cloud

This is an excerpt from Manning's book Designing Cloud Data Platforms MEAP V06. Login to get full access to this book.

We will start by describing two potential architectures: one that is centered around a cloud data warehouse only and another one that uses broader design principles to define a data platform. Then we will walk through examples showing how to load and work with the data in both solutions. We will specifically focus on what happens to the data platform pipelines when there are changes to source data structure and look at how data platform architecture can help you analyze semi structured data at scale. As similar outcomes can be achieved by directly ingesting data into the cloud data warehouse, we’ll also walk through loading and working with the same data in a data warehouse alone.

Figure 2.14. The data warehouse becomes just another component in a data platform architecture

As the data ingestion layer usually doesn’t store any data itself, though it may use transient cache, once data is passed through the ingestion layer it must be stored reliably. The storage layer in the data platform architecture is responsible for persisting data for long term consumption. It has two types of storage - Fast and Slow, as shown on the diagram below.