Model Serving

As explained in Designing Deep Learning Systems: A guide for software engineers

A production service that hosts a machine learning model and accepts requests to generate predictions.

Model serving hosts a trained machine learning model, accepts input data, executes the model, and returns predictions. A common form is a prediction service: a web service exposing a model’s predict function through an API, while hiding model loading and framework-specific execution from callers. section 6.1.3 section 6.1.5

A typical request path receives a prediction request, loads the required model from an artifact store into memory or GPU, runs its algorithm, and returns a result. Serving may support online, low-latency requests or offline batch inference; requirements determine network placement, model count, hardware, memory, and supporting preprocessing components. section 1.2.2 section 6.1.3

Figure 6.3 A typical model prediction workflow in a prediction service

Follow the basic relationship among the prediction API, prediction service, and model artifact store. section 6.1.3

Choose an architecture from the user scenario rather than aiming for one universal service:

• Use direct model embedding when prediction runs locally inside an application. section 6.2
• Use a model service for one model, version, or model type. Package execution logic in a container, expose HTTP or gRPC, and scale instances behind a load balancer. This is simple, but separate models create separate services to patch, deploy, and monitor. section 6.2.2
• Use a model service with an in-memory model cache for multitenant workloads whose tenant models share an algorithm and prediction function. section 6.3.2
• Use a centralized model server or prediction platform when multiple applications need different model types. It consolidates onboarding and maintenance but requires unified APIs, routing, versioning, caching, execution support, and model management. section 6.3.3 section 6.4

Figure 6.7 A general prediction service (platform) design that works with arbitrary model types

Inspect the components needed when one platform must route and execute arbitrary model types. section 6.3.3

Tenant-specific chatbot predictions

A multitenant chatbot can select an intent-classification model for each tenant. Because the models use the same algorithm, one prediction service can keep model graphs and dependent data in a cache, loading model files as needed. section 6.3.2

Production serving must balance latency with resource saturation and accommodate differing input and output formats, frameworks, model sizes, and hardware needs. Treat deployment safety, rollback, monitoring, and alerting as core requirements: outages or rising latency immediately affect applications that depend on real-time predictions. section 6.1.4 section 6.3.4

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Explains request workflow, architecture choices, multitenant caching, and platform components, while deferring concrete service implementations and tool demonstrations.

Examples

Uses shark-image detection, document classification, speech translation, and a multitenant intent-classification chatbot.

What it assumes

Assumes some distributed-computing experience while defining serving terminology and core prediction workflow from zero.

What you can do after

Operate/configure it by selecting model service, cached multitenant service, or prediction platform designs for stated workloads.

How it connects

Connects model serving to model deployment, artifact stores, APIs, model caching, monitoring, alerting, and MLOps.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Distributed Machine Learning PatternsA software system that receives prediction requests and returns outputs from a deployed machine learning model.
• 

  liveBookmanning.com
  Delivering Machine Learning Projects: From design to deploymentA model-serving system is the software and infrastructure that loads trained machine learning models and handles prediction requests in an operating environment.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringA model serving framework is software that packages, exposes, operates, and scales machine learning models for inference.
• 

  liveBookmanning.com
  Generative AI in ActionModel serving is the provision of a trained machine-learning model as an operational service that accepts inputs and returns predictions or generated outputs.

Compare how 5 books explain Model Serving →

Related concepts

• generationequivalent to · Generation is the computational production of outputs from supplied inputs or context, often by applying a model to produce predictions or new content.
• NVIDIA Triton Inference Serviceincludes · NVIDIA Triton Inference Server is open-source software for deploying and serving machine-learning models through standardized interfaces across frameworks and hardware backends.
• language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• KServeuses · KServe is an open-source Kubernetes-native platform for deploying and serving machine-learning models through standardized resources and interfaces.
• endpointproduces · An endpoint is a network-accessible address or interface through which a service exposes a resource or operation and handles requests and responses.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.