liveBook · Manning

Model Serving

Also called inference server, inference service, model server

Model serving is the software infrastructure that hosts trained machine-learning models and handles inference requests, returning model outputs.

How 6 Manning books explain it

Designing Deep Learning Systems: A guide for software engineers

A production service that hosts a machine learning model and accepts requests to generate predictions.

How it is used
Delivers trained-model predictions to applications through web APIs after models move into production.
How deep it goes
Explains request workflow, architecture choices, multitenant caching, and platform components, while deferring concrete service implementations and tool demonstrations.
Examples
Uses shark-image detection, document classification, speech translation, and a multitenant intent-classification chatbot.
What you can do after
Operate/configure it by selecting model service, cached multitenant service, or prediction platform designs for stated workloads.

Read how Designing Deep Learning Systems: A guide for software engineers explains Model Serving →

Distributed Machine Learning Patterns

A software system that receives prediction requests and returns outputs from a deployed machine learning model.

How it is used
Delivers predictions from trained models as the user-facing stage of a distributed machine-learning pipeline.
How deep it goes
Defines serving, contrasts single-machine and clustered deployments, and develops replication, sharding, and event-driven resource patterns.
Examples
Uses video entity tagging, hotel-price prediction traffic, and Fashion-MNIST image classification to illustrate serving designs.
What you can do after
Operate/configure it with replication, sharding, shared resources, autoscaling, and rate limits for serving workloads.

Read how Distributed Machine Learning Patterns explains Model Serving →

Delivering Machine Learning Projects: From design to deployment

A model-serving system is the software and infrastructure that loads trained machine learning models and handles prediction requests in an operating environment.

How it is used
Connects selected models to production data and application decisions as part of system delivery.
How deep it goes
Defines inference services and compares execution approaches and tradeoffs without detailing a server implementation.
Examples
Uses a shopping-basket event and large-scale translation and recommendation services to illustrate inputs and execution scale.
What you can do after
Evaluate serving approaches against latency, request volume, cost, and production integration requirements.

Read how Delivering Machine Learning Projects: From design to deployment explains Model Serving →

Machine Learning Platform Engineering

A model serving framework is software that packages, exposes, operates, and scales machine learning models for inference.

How it is used
Deploys trained models as APIs or batch inference workloads within the production ML life cycle.
How deep it goes
Builds a runnable service through packaging, local testing, containerization, and deployment choices, but defers infrastructure internals.
Examples
Uses recommender and object-detection services, MLflow production aliases, /predict requests, Bento files, Docker containers, and local inference servers.
What you can do after
Apply it via BentoML or MLflow to package, expose, validate, and containerize deployable inference services.

Read how Machine Learning Platform Engineering explains Model Serving →

Generative AI in Action

Model serving is the provision of a trained machine-learning model as an operational service that accepts inputs and returns predictions or generated outputs.

How it is used
Deploys trained models for responsive, scalable enterprise applications and production inference.
How deep it goes
Explains infrastructure, latency, scaling, caching, load balancing, updates, and observability without descending into model internals.
Examples
Uses MLflow and Prometheus deployment configuration, inference metric tracking, and production update scenarios as concrete examples.
What you can do after
Operate/configure serving infrastructure with scaling, caching, load balancing, observability, quotas, and controlled model updates.

Read how Generative AI in Action explains Model Serving →

CUDA for LLMs

Model serving is the operation of a deployed model to process inference requests and return results.

How it is used
Coordinates deployed-model inference as an end-to-end service rather than treating a forward pass or fast kernel as the whole job.
How deep it goes
Enumerates serving components and system bottlenecks, but stops short of prescribing production integration.
Examples
Uses time-to-first-token, intertoken latency, target-latency throughput, and Nsight Systems profiling as operational examples.
What you can do after
Diagnose serving bottlenecks by profiling a model end to end with Nsight Systems and evaluating system-level metrics.

Read how CUDA for LLMs explains Model Serving →

Also covered in

What to know first

Related concepts

Questions about Model Serving

What is Model Serving?

Model serving is the software infrastructure that hosts trained machine-learning models and handles inference requests, returning model outputs.

Which Manning books explain Model Serving?

Designing Deep Learning Systems: A guide for software engineers, Distributed Machine Learning Patterns, Delivering Machine Learning Projects: From design to deployment, Machine Learning Platform Engineering, Generative AI in Action and CUDA for LLMs.

What should I know before learning Model Serving?

Manning books that teach Model Serving usually introduce language model, production, generation and pipeline first.

What kinds of Model Serving are there?

Kinds of Model Serving include Triton Inference Server.

What are the parts of Model Serving?

Model Serving includes model loading, batching, request scheduling and token streaming.

What is Model Serving also called?

Model Serving is also called inference server, inference service or model server.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.