Model Serving
Model serving is the software infrastructure that hosts trained machine-learning models and handles inference requests, returning model outputs.
Designing Deep Learning Systems: A guide for software engineers
A production service that hosts a machine learning model and accepts requests to generate predictions.
- How it is used
- Delivers trained-model predictions to applications through web APIs after models move into production.
- How deep it goes
- Explains request workflow, architecture choices, multitenant caching, and platform components, while deferring concrete service implementations and tool demonstrations.
- Examples
- Uses shark-image detection, document classification, speech translation, and a multitenant intent-classification chatbot.
- What you can do after
- Operate/configure it by selecting model service, cached multitenant service, or prediction platform designs for stated workloads.
Read how Designing Deep Learning Systems: A guide for software engineers explains Model Serving →
Distributed Machine Learning Patterns
A software system that receives prediction requests and returns outputs from a deployed machine learning model.
- How it is used
- Delivers predictions from trained models as the user-facing stage of a distributed machine-learning pipeline.
- How deep it goes
- Defines serving, contrasts single-machine and clustered deployments, and develops replication, sharding, and event-driven resource patterns.
- Examples
- Uses video entity tagging, hotel-price prediction traffic, and Fashion-MNIST image classification to illustrate serving designs.
- What you can do after
- Operate/configure it with replication, sharding, shared resources, autoscaling, and rate limits for serving workloads.
Read how Distributed Machine Learning Patterns explains Model Serving →
Delivering Machine Learning Projects: From design to deployment
A model-serving system is the software and infrastructure that loads trained machine learning models and handles prediction requests in an operating environment.
- How it is used
- Connects selected models to production data and application decisions as part of system delivery.
- How deep it goes
- Defines inference services and compares execution approaches and tradeoffs without detailing a server implementation.
- Examples
- Uses a shopping-basket event and large-scale translation and recommendation services to illustrate inputs and execution scale.
- What you can do after
- Evaluate serving approaches against latency, request volume, cost, and production integration requirements.
Read how Delivering Machine Learning Projects: From design to deployment explains Model Serving →
Machine Learning Platform Engineering
A model serving framework is software that packages, exposes, operates, and scales machine learning models for inference.
- How it is used
- Deploys trained models as APIs or batch inference workloads within the production ML life cycle.
- How deep it goes
- Builds a runnable service through packaging, local testing, containerization, and deployment choices, but defers infrastructure internals.
- Examples
- Uses recommender and object-detection services, MLflow production aliases, /predict requests, Bento files, Docker containers, and local inference servers.
- What you can do after
- Apply it via BentoML or MLflow to package, expose, validate, and containerize deployable inference services.
Read how Machine Learning Platform Engineering explains Model Serving →
Generative AI in Action
Model serving is the provision of a trained machine-learning model as an operational service that accepts inputs and returns predictions or generated outputs.
- How it is used
- Deploys trained models for responsive, scalable enterprise applications and production inference.
- How deep it goes
- Explains infrastructure, latency, scaling, caching, load balancing, updates, and observability without descending into model internals.
- Examples
- Uses MLflow and Prometheus deployment configuration, inference metric tracking, and production update scenarios as concrete examples.
- What you can do after
- Operate/configure serving infrastructure with scaling, caching, load balancing, observability, quotas, and controlled model updates.
CUDA for LLMs
Model serving is the operation of a deployed model to process inference requests and return results.
- How it is used
- Coordinates deployed-model inference as an end-to-end service rather than treating a forward pass or fast kernel as the whole job.
- How deep it goes
- Enumerates serving components and system bottlenecks, but stops short of prescribing production integration.
- Examples
- Uses time-to-first-token, intertoken latency, target-latency throughput, and Nsight Systems profiling as operational examples.
- What you can do after
- Diagnose serving bottlenecks by profiling a model end to end with Nsight Systems and evaluating system-level metrics.
Also covered in
- Kafka for Architects
- Rearchitecting LLMsIn early access (MEAP)
- Machine Learning System Design
- Self-Hosting LLMs at ScaleIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- productionProduction is the live operational setting in which a deployed software system serves its intended users or processes real workloads.
- generationGeneration is the production of an output by a computational model from supplied input or context, often using patterns learned from data.
- pipelineA pipeline is an ordered sequence of connected processing stages through which data or work flows, with each stage passing or transforming its output for subsequent stages.
Related concepts
- batchinghas part · Batching is the grouping of multiple items, requests, or operations so they can be processed together as a unit.
- Triton Inference Serverincludes · NVIDIA Triton Inference Server is open-source software for deploying and serving machine-learning models across multiple frameworks and hardware backends.
- generationequivalent to · Generation is the production of an output by a computational model from supplied input or context, often using patterns learned from data.
- token streaminghas part · Token streaming is the incremental delivery of generated output as tokens become available, rather than waiting for the complete response.
Questions about Model Serving
What is Model Serving?
Model serving is the software infrastructure that hosts trained machine-learning models and handles inference requests, returning model outputs.
Which Manning books explain Model Serving?
Designing Deep Learning Systems: A guide for software engineers, Distributed Machine Learning Patterns, Delivering Machine Learning Projects: From design to deployment, Machine Learning Platform Engineering, Generative AI in Action and CUDA for LLMs.
What should I know before learning Model Serving?
Manning books that teach Model Serving usually introduce language model, production, generation and pipeline first.
What kinds of Model Serving are there?
Kinds of Model Serving include Triton Inference Server.
What are the parts of Model Serving?
Model Serving includes model loading, batching, request scheduling and token streaming.
What is Model Serving also called?
Model Serving is also called inference server, inference service or model server.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





