Pipeline focus

As explained in Designing Deep Learning Systems: A guide for software engineers

A metadata-system design approach that organizes and correlates metadata primarily around pipelines or training runs.

Pipeline focus organizes metadata and artifacts around a pipeline or training run rather than around model files. A shared training_run_id links the run to its output model, input data, metrics, and artifact URL, allowing navigation from execution to results. section 8.3.2

A pipeline-focused storage layer supports practical retrieval patterns:

• Preserve test, preproduction, and production entities under their associated training run. section 8.3.2
• Store the output artifact URL on the Training_Runs object and retrieve it through model or execution queries. section 8.3.2
• Add web-interface search for both pipeline and model views; pipeline focus and model focus are complementary rather than mutually exclusive. section 8.3.2

Figure 8.5 An entity relationship diagram of the data schema of the metadata storage

Use the entity-relationship schema to trace training_run_id links among executions, models, data, metrics, and artifact locations. section 8.3.2

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Details shared identifiers, entity relationships, artifact URLs, and dual search, without specifying an implementation technology.

Examples

Uses test, preproduction, and production entities plus linked models, input data, metrics, and artifact URLs.

What it assumes

Assumes familiarity with metadata entities, training runs, models, artifacts, and search interfaces.

What you can do after

Evaluate/understand it by tracing a training run to its model, data, metrics, and stored artifact.

How it connects

Connects metadata stores with training runs, artifact stores, model focus, pipeline search, and model search.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Related concepts

• model focuscontrasts with · A metadata-system design approach that organizes and correlates metadata primarily around model files.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.