14 Data Mesh revolutionizing data engineering

If you thought modernizing your architecture would not affect your data, you would be seriously wrong. I won't try to convince you of the importance of data; anyone who has read this far into this chapter knows already. Many people nicknamed data the new oil, but modern data engineering goes beyond simple pipelines. Data feeds everything from dashboards and reports executives use to make decisions, to risk analysis and fraud detection, including AI... But unleashing the true value of data comes at a severe operational cost if not done correctly. For health reasons, I will not name the organizations whose operating budget for maintaining pipelines and systems forbids them to do any forward thinking. Then, they hire many data scientists who spend 80% of their time on data discovery and engineering. Finally, most complain about the value data brings to the company. Sounds familiar?

In this chapter, I will walk you through how we came to this point, what are issues with modern data management, why you can solve them with only four fundamental principles, the various elements of the architecture, and finally, how to get started.

14.1 Setting up the context for complex data

14.1.1 The dawn of data engineering

14.1.2 New needs around data

14.1.3 More problems than solutions

14.2 The four principles of Data Mesh

14.2.1 Principle of Domain Ownership

14.2.2 Principle of Data as a Product

14.2.3 Principle of the Self-Serve Data Platform

14.2.4 Principle of Federated Computational Governance

14.2.5 No principle lives in isolation

14.3 Building your first data quantum

14.3.1 The smallest element with value

14.3.2 Logical architecture

14.3.3 Your new best friend: the data contract

14.3.4 Physical architecture

14.4 Navigating through the planes

14.4.1 The infrastructure experience plane

14.4.2 The data product experience plane

14.4.3 The mesh experience plane

14.5 First & next steps

14.6 Summary