liveBook · Manning

Data aggregator

As explained in Designing Deep Learning Systems: A guide for software engineers

A software component that organizes related metadata and routes it to storage or query results.

A data aggregator is the organizing layer inside a metadata store server. It understands how metadata is structured and linked, routes new records to the appropriate storage locations, and serves metadata-search queries. section 8.3.2

Figure 8.4 A general metadata and artifact store system design

The reference architecture places the data aggregator between a RESTful web interface and an abstract storage layer, alongside a client SDK and web UI. section 8.3.2

Use the aggregator to:

  • Add metadata according to relationships among training runs, parameters, metrics, and model artifacts. section 8.3.2
  • Serve searches without exposing whether records reside in cloud object storage, local files, SQL, or NoSQL systems. section 8.3.2
Figure 8.5 An entity relationship diagram of the data schema of the metadata storage

The entity relationship diagram helps trace how training executions connect to parameters, metrics, and produced model files. section 8.3.2

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes
Defines responsibilities and schema relationships but stops short of implementation details for aggregation algorithms.
Examples
Uses training runs linked to dataset identifiers, versions, hyperparameters, performance metrics, and produced model files.
What you can do after
Evaluate/understand where metadata organization and query routing belong in a storage-backend-independent architecture.
How it connects
Connects RESTful interfaces, client SDKs, storage adapters, training runs, parameters, metrics, and model artifacts.
Cover of Designing Deep Learning Systems: A guide for software engineers
liveBookmanning.com
Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.