Model Serving

As explained in Distributed Machine Learning Patterns

A software system that receives prediction requests and returns outputs from a deployed machine learning model.

Model serving loads a previously trained machine learning model and uses it to generate predictions or inferences from new input data. It follows training in the machine-learning pipeline and is the component whose results reach users directly. section 4.1 section 9.3

Figure 4.1 A diagram showing where model serving fits in the machine learning pipeline

Shows model serving as the pipeline stage after a model has been trained. section 4.1

A small deployment can run as one local program with a model and dataset that fit on one machine. Distributed serving instead runs across a cluster, where models and data may reside in a remote distributed database or across multiple disks. The distributed design must remain scalable, reliable, and efficient as request volume and request size grow. section 4.1 section 4.0

Video entity tagging

Users upload new videos to a stateless service. A model server retrieves a trained entity-tagging model from model storage, processes each video, and returns possible themes such as visual entities. A single server handling requests first-come, first-served creates long waits as traffic rises. section 4.2

Choose a serving pattern according to the operational bottleneck:

• Use replicated services to handle a growing number of requests through horizontal scaling. section 4.9
• Use sharded services when one large request cannot fit in the memory of an individual replica; distribute its processing across multiple shards. section 4.3.1 section 4.9
• Use event-driven processing when traffic is uncertain. Draw from a shared resource pool, pre-allocate for historical peaks, autoscale at limits, and apply rate limits to control concurrent requests. section 4.4 section 4.4.2

Resource allocation is a trade-off. Fixed capacity can leave resources idle when holiday-price prediction traffic falls, yet an unexpected surge can exceed planned capacity. A shared pool lets serving borrow resources needed for a surge while other pipeline components use the same pool; separate pools may be preferable when training needs different resources, such as GPUs. section 4.4 section 4.4.2

Serving in an end-to-end image-classification workflow

After data ingestion, distributed training, and model selection, the selected model is deployed for serving; predictions are then aggregated for users. One implementation creates a KServe InferenceService for a TensorFlow model from an Argo Workflow, with trained artifacts persisted for the serving step. section 7.1.2 section 9.4.1

How Distributed Machine Learning Patterns covers it

How deep it goes

Defines serving, contrasts single-machine and clustered deployments, and develops replication, sharding, and event-driven resource patterns.

Examples

Uses video entity tagging, hotel-price prediction traffic, and Fashion-MNIST image classification to illustrate serving designs.

What it assumes

Assumes a previously trained model and situates serving after training within a machine-learning pipeline.

What you can do after

Operate/configure it with replication, sharding, shared resources, autoscaling, and rate limits for serving workloads.

How it connects

Connects model serving with model training, model selection, data ingestion, replicated services, sharded services, and event-driven processing.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Designing Deep Learning Systems: A guide for software engineersA production service that hosts a machine learning model and accepts requests to generate predictions.
• 

  liveBookmanning.com
  Delivering Machine Learning Projects: From design to deploymentA model-serving system is the software and infrastructure that loads trained machine learning models and handles prediction requests in an operating environment.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringA model serving framework is software that packages, exposes, operates, and scales machine learning models for inference.
• 

  liveBookmanning.com
  Generative AI in ActionModel serving is the provision of a trained machine-learning model as an operational service that accepts inputs and returns predictions or generated outputs.

Compare how 5 books explain Model Serving →

Related concepts

• generationequivalent to · Generation is the computational production of outputs from supplied inputs or context, often by applying a model to produce predictions or new content.
• NVIDIA Triton Inference Serviceincludes · NVIDIA Triton Inference Server is open-source software for deploying and serving machine-learning models through standardized interfaces across frameworks and hardware backends.
• language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• KServeuses · KServe is an open-source Kubernetes-native platform for deploying and serving machine-learning models through standardized resources and interfaces.
• endpointproduces · An endpoint is a network-accessible address or interface through which a service exposes a resource or operation and handles requests and responses.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.