liveBook · Manning

Data architecture

As explained in Delivering Machine Learning Projects: From design to deployment

A corporate data architecture is an organization's established structure for storing, integrating, governing, and processing business data.

Data architecture is the established structure used to store, integrate, govern, and process business data. For machine learning systems, it may require different infrastructure from standard corporate data architecture because workloads can affect other applications. section 4.4

Treat architecture as a delivery artifact:

  • Create an infrastructure ticket to build and share the infrastructure plan. section 4.4
  • Use a pragmatic design when existing corporate systems constrain available options. section 4.4
  • Trace datasets back through real-time databases, data warehouses, data lakes, and other stores. section 4.5.3
  • Record lineage in a data catalogue, lineage system, metadata-management process, or master data model. section 4.5.3

Tracing a dataset

A table used for analytics may be several transformations removed from questions, logs, or sensor readings. A lineage system such as Alation or Collibra can provide a map of its origin when the surrounding data estate has become difficult to decode. section 4.5.3

How Delivering Machine Learning Projects: From design to deployment covers it

How deep it goes
Explains infrastructure planning and lineage practices, but stops short of specifying architectural designs or automated tracing implementations.
Examples
Uses infrastructure tickets, data warehouses, data lakes, real-time databases, master data models, Alation, and Collibra as concrete artifacts or systems.
What it assumes
Assumes an existing corporate data estate and introduces the need to distinguish machine learning infrastructure from standard enterprise workloads.
What you can do after
Diagnose dataset origins and plan infrastructure that reduces disruption when machine learning workloads share corporate systems.
How it connects
Links data architecture to machine learning systems, infrastructure design, data lineage, data catalogues, metadata management, data warehouses, and data lakes.
Cover of Delivering Machine Learning Projects: From design to deployment
liveBookmanning.com
Read Delivering Machine Learning Projects: From design to deployment on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Delivering Machine Learning Projects: From design to deployment. The section links point to the parts of the book it draws on.