Model Serving
A model-serving platform is software infrastructure that hosts trained models and exposes inference by accepting requests and returning model outputs.
Designing Deep Learning Systems: A guide for software engineers
A production service that hosts a machine learning model and accepts requests to generate predictions.
- How it is used
- Delivers trained-model predictions to applications through web APIs after models move into production.
- How deep it goes
- Explains request workflow, architecture choices, multitenant caching, and platform components, while deferring concrete service implementations and tool demonstrations.
- Examples
- Uses shark-image detection, document classification, speech translation, and a multitenant intent-classification chatbot.
- What you can do after
- Operate/configure it by selecting model service, cached multitenant service, or prediction platform designs for stated workloads.
Read how Designing Deep Learning Systems: A guide for software engineers explains Model Serving →
Distributed Machine Learning Patterns
A software system that receives prediction requests and returns outputs from a deployed machine learning model.
- How it is used
- Delivers predictions from trained models as the user-facing stage of a distributed machine-learning pipeline.
- How deep it goes
- Defines serving, contrasts single-machine and clustered deployments, and develops replication, sharding, and event-driven resource patterns.
- Examples
- Uses video entity tagging, hotel-price prediction traffic, and Fashion-MNIST image classification to illustrate serving designs.
- What you can do after
- Operate/configure it with replication, sharding, shared resources, autoscaling, and rate limits for serving workloads.
Read how Distributed Machine Learning Patterns explains Model Serving →
Delivering Machine Learning Projects: From design to deployment
A model-serving system is the software and infrastructure that loads trained machine learning models and handles prediction requests in an operating environment.
- How it is used
- Turns selected models into a production application capability that consumes operational data and supplies outputs for application decisions.
- How deep it goes
- Defines the inference service, traces its production inputs and outputs, and compares execution options without detailing model-server code.
- Examples
- Uses batch files, application events, a development-environment prototype, and a table of generic execution engines.
- What you can do after
- Build it from scratch as an integrated production inference path, then test and hand it over for ongoing support.
Read how Delivering Machine Learning Projects: From design to deployment explains Model Serving →
Machine Learning Platform Engineering
A model serving framework is software that packages, exposes, operates, and scales machine learning models for inference.
- How it is used
- Deploys trained models as APIs or batch inference workloads within the production ML life cycle.
- How deep it goes
- Builds a runnable service through packaging, local testing, containerization, and deployment choices, but defers infrastructure internals.
- Examples
- Uses recommender and object-detection services, MLflow production aliases, /predict requests, Bento files, Docker containers, and local inference servers.
- What you can do after
- Apply it via BentoML or MLflow to package, expose, validate, and containerize deployable inference services.
Read how Machine Learning Platform Engineering explains Model Serving →
Generative AI in Action
Model serving is the provision of a trained machine-learning model as an operational service that accepts inputs and returns predictions or generated outputs.
- How it is used
- Deploys trained models for responsive, scalable enterprise applications and production inference.
- How deep it goes
- Explains infrastructure, latency, scaling, caching, load balancing, updates, and observability without descending into model internals.
- Examples
- Uses MLflow and Prometheus deployment configuration, inference metric tracking, and production update scenarios as concrete examples.
- What you can do after
- Operate/configure serving infrastructure with scaling, caching, load balancing, observability, quotas, and controlled model updates.
Related concepts
- generationequivalent to · Generation is the computational production of outputs from supplied inputs or context, often by applying a model to produce predictions or new content.
- NVIDIA Triton Inference Serviceincludes · NVIDIA Triton Inference Server is open-source software for deploying and serving machine-learning models through standardized interfaces across frameworks and hardware backends.
- language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- KServeuses · KServe is an open-source Kubernetes-native platform for deploying and serving machine-learning models through standardized resources and interfaces.
- endpointproduces · An endpoint is a network-accessible address or interface through which a service exposes a resource or operation and handles requests and responses.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.