Training pipeline

As explained in Designing Deep Learning Systems: A guide for software engineers

An automated workflow that executes model-training tasks from input data through model production, possibly including deployment.

A training pipeline organizes model development from input data and training code to a produced model. An experimental pipeline supports development and testing; an automated production pipeline runs on customer data and can deploy models directly to a prediction service. section 8.2.2

A production-oriented pipeline connects:

• Data collection and parsing as input preparation. section 8.2.2
• Training algorithms that produce intent classification models. section 8.2.2
• Automatic model deployment to a prediction service. section 8.2.2

Chatbot intent models

A team first builds and tests an experimental intent-model pipeline, then promotes the algorithm to production. When a new dataset causes a customer’s accuracy to drop, investigators trace the deployed model to its dataset and code versions and reproduce it for debugging. section 8.2.2

Figure 8.2 Model development without metadata management

Use the workflow figure to compare model development with and without metadata management. section 8.2.2

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Distinguishes experimentation from production execution and deployment, without detailing pipeline implementation.

Examples

Uses chatbot intent models, customer data, prediction services, and a dataset-induced accuracy regression.

What you can do after

Builds and deploys models automatically, then reproduces and compares versions during troubleshooting.

How it connects

Connects data collection, training algorithms, metadata management, model lineage, and model deployment.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.