14 Data mesh revolutionizing data engineering

 

This chapter covers

  • Quickly reviewing why and how we got into this situation
  • Anchoring the four principles of data mesh
  • Understanding data quantum
  • Building your first data quantum
  • Assimilating the data contract
  • Navigating through the experience planes

If you thought modernizing your architecture would not affect your data, you would be seriously wrong. I won’t try to convince you of the importance of data; anyone who has read this far into this chapter knows already. Many people nicknamed data the new oil, but modern data engineering goes beyond simple pipelines. Data feeds everything from dashboards and reports used by executives when making decisions to risk analysis and fraud detection, including AI. However, unleashing the true value of data comes at a severe operational cost if not done correctly. For sanity’s sake, I will not name the organizations whose operating budget for maintaining pipelines and systems forbids them to do any forward-thinking; then, they hire many data scientists who spend 80% of their time on data discovery and engineering. Finally, most complain about the value data brings to the company. Sound familiar? In this chapter, I will walk you through how we came to this point, what the issues are with modern data management, why you can solve them with just four fundamental principles, what the various elements of the architecture are, and finally, how to get started.

14.1 Setting up the context for complex data

14.1.1 The dawn of data engineering

14.1.2 New needs around data

14.1.3 More problems than solutions

14.2 The four principles of data mesh

14.2.1 Principle of domain ownership

14.2.2 Principle of data as a product

14.2.3 Principle of the self-serve data platform

14.2.4 Principle of federated computational governance

14.2.5 No principle lives in isolation

14.3 Building your first data quantum

sitemap