liveBook · Manning

Model manager

As explained in Designing Deep Learning Systems: A guide for software engineers

A software component that loads, initializes, caches, and executes machine learning models.

A model manager is the execution core of a prediction service. It loads model files, initializes models, caches them in memory, and runs inference on user input. In a typical architecture, a frontend sends a model identifier and document to a gRPC predictor, which delegates model handling to the manager. section 7.1.4

A practical model-manager workflow includes:

  • Load vocabulary, class mappings, manifests, and model weights from a model directory or shared disk volume. section 7.1.4
  • Preprocess request data, execute the selected model, and postprocess the result before returning it. section 7.1.4

Intent prediction service

A Python predictor constructs a ModelManager with configuration, a tokenizer, and a CPU device, then injects it into a gRPC service. Each request supplies a runId and document; the service loads the corresponding model and asks the manager to predict. In a larger deployment, a thin model-serving layer can query metadata and artifact stores to identify the correct inference container and model file. section 10.2.5

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes
Implements cache checks, artifact loading, model initialization, and prediction, while leaving broader production management details unspecified.
Examples
Uses an intent-classification predictor, shared disk artifacts, and a production visual-recognition inference setup.
What it assumes
Assumes prediction-service components and model artifacts, while demonstrating the loading-to-inference workflow explicitly.
What you can do after
Apply it via a model-serving service that loads, caches, and executes models for inference requests.
How it connects
Connects model serving with gRPC, model artifacts, metadata stores, preprocessing, and inference.
Cover of Designing Deep Learning Systems: A guide for software engineers
liveBookmanning.com
Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.