Model Serving

As explained in Delivering Machine Learning Projects: From design to deployment

A model-serving system is the software and infrastructure that loads trained machine learning models and handles prediction requests in an operating environment.

A trained model does not deliver a service merely by sitting in a repository. Model serving puts it into an application’s operating environment: production data flows supply the inputs, an inference service runs the model, and application code uses its output to decide what happens next. The service may process files in batches or respond to application events. section 9.0 section 9.1

Figure 9.4 An abstract view of the production ML implementation. Data is pulled into and collected in a feature creation process, and then it’s submitted to a running instance of the model(s) that are used to generate an inference and an output.

The production path joins data collection, feature creation, a running model, and its resulting output. section 9.4.1

Treat serving as a production delivery task, not a model-only deployment:

• Build data flows that reproduce the features and formats expected by the selected model. Production hardware and systems may differ from those used in development and testing. section 9.4.1
• Build the model server and inference system against the project’s latency, request-volume, and overall cost requirements. section 9.4 section 9.4.2
• Connect outputs to the application and its interfaces; provide logging, monitoring, alerts, and integration signals needed to operate the service. section 9.1 section 9.11

Table 9.5 Using generic execution engines for ML models (view table figure)

Engine	Comments
In a database: BigQuery ML, Redshift ML, Oracle OCI, Vertica PMML	Execution in database functions means accepting constraints on how the application runs, but this is a modular and decoupled approach.
Application servers: NGINX, Apache Tomcat, Flask, Appserver	For generic low-scale and permissive latency/cost use cases. Consider because of the low barrier in terms of skills and setup requirements.
SPARK-like solutions: Apache SPARK, Databricks, Dataproc	Good support for bulk and stream processing with permissive cost and latency requirements.
Kubernetes, Kubeflow	Flexible and scalable support for interactive applications with permissive cost requirements.
Serverless: Lambda, Cloud Functions, Azure Functions, Cloud Run.	A flexible, modular, and decoupled approach that’s cost effective and performative in some applications but can become expensive in unpredictable ways.

Compare generic execution engines—including databases, application servers, stream-processing platforms, Kubernetes-based platforms, and serverless services—against the workload rather than selecting one by default. section 9.4.2

Prototype before committing to the production design

A delivery team can split a development-environment prototype across the data pipeline, model-server design, and user interface, then review the assembled application with stakeholders. This exposes architectural questions while the production design is still being worked out. section 9.10

Before release, test the integrated system, obtain production sign-off, and agree who will govern and support it. Model serving remains an operational responsibility after users receive the system. section 9.1 section 9.11

How Delivering Machine Learning Projects: From design to deployment covers it

How deep it goes

Defines the inference service, traces its production inputs and outputs, and compares execution options without detailing model-server code.

Examples

Uses batch files, application events, a development-environment prototype, and a table of generic execution engines.

What it assumes

Assumes trained, tested, selected models and earlier training and test pipelines; defines the inference service before discussing its production choices.

What you can do after

Build it from scratch as an integrated production inference path, then test and hand it over for ongoing support.

How it connects

Connects serving to data pipelines, feature creation, application interfaces, latency, monitoring, and production governance.

liveBookmanning.com

Read Delivering Machine Learning Projects: From design to deployment on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Designing Deep Learning Systems: A guide for software engineersA production service that hosts a machine learning model and accepts requests to generate predictions.
• 

  liveBookmanning.com
  Distributed Machine Learning PatternsA software system that receives prediction requests and returns outputs from a deployed machine learning model.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringA model serving framework is software that packages, exposes, operates, and scales machine learning models for inference.
• 

  liveBookmanning.com
  Generative AI in ActionModel serving is the provision of a trained machine-learning model as an operational service that accepts inputs and returns predictions or generated outputs.

Compare how 5 books explain Model Serving →

Related concepts

• generationequivalent to · Generation is the computational production of outputs from supplied inputs or context, often by applying a model to produce predictions or new content.
• NVIDIA Triton Inference Serviceincludes · NVIDIA Triton Inference Server is open-source software for deploying and serving machine-learning models through standardized interfaces across frameworks and hardware backends.
• language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• KServeuses · KServe is an open-source Kubernetes-native platform for deploying and serving machine-learning models through standardized resources and interfaces.
• endpointproduces · An endpoint is a network-accessible address or interface through which a service exposes a resource or operation and handles requests and responses.

This explainer was generated by Manning's knowledge graph from the text of Delivering Machine Learning Projects: From design to deployment. The section links point to the parts of the book it draws on.