Chapter 10: Model inference and serving
The chapter introduces the move from trained, validated models to serving them as APIs, outlines the BentoML deployment path, and explains the value and limits of self-service deployment.
After this chapter you will be able to
- Path from models to services. Describe how the chapter moves from the trained object detection and movie recommendation models to services: first building and running a BentoML service locally, then packaging it in a container for deployment. Mention that the chapter also considers using BentoML with MLflow and alternatives.
- Self-service deployment tradeoffs. Explain why self-service deployment helps model developers test and iterate with fewer manual errors and bottlenecks, while distinguishing nonproduction testing by data scientists from production deployment by ML engineers.
- ML deployment complexity. Explain how ML deployment builds on conventional software practices such as testing and CI/CD but adds model-specific concerns that require specialized tools and collaboration between data scientists and ML engineers.
- Inference and infrastructure choices. Describe how the intended inference pattern—batch predictions, real-time inference, or both—affects deployment, alongside variable workloads, compute-resource allocation, and possible GPU requirements.
- Model reliability over time. Explain why serving must preserve required input formats and preprocessing, monitor and log model behavior, and manage retraining, new versions, and rollback.
- Unified and reproducible serving. Explain how BentoML simplifies deployment by providing a common way to package and serve models from different frameworks. Describe how bundling the model, dependencies, and serving logic makes builds reproducible, while a high-level API reduces serving-code boilerplate.
- YOLOv8 service deployment target. Describe the plan to deploy the YOLOv8 model trained in the previous chapter as an ML service using a BentoML Service.
- Service components and request flow. Trace an inference request through a BentoML Service: an API server receives and validates the HTTP input, then passes arguments to a Runner that wraps the model and performs inference.
- Object detection endpoint outputs. Distinguish the planned YOLOv8 service endpoints: /inference returns JSON results for an input image, whereas /render returns an image annotated with bounding box, label, and probability. Explain the programmatic versus visual uses of those outputs.
- Local BentoML serving plan. Describe the planned local workflow: start with a Runner, create and run a BentoML Service, and package the object detection model for access through a REST API during testing and development.
- YOLOv8 Runnable and Runner. Describe how the service loads the previous chapter’s YOLOv8 weights in a Runnable, exposes an inference method, and creates a named Runner from that Runnable.
- Asynchronous image inference endpoint. Trace an image request through the BentoML Service API to the Runner and explain why the endpoint awaits the asynchronous inference result.
- Downloadable rendered detection image. Describe how the /render endpoint saves an inference image with bounding boxes and labels and returns it for download.
- Built-in service observability endpoints. Distinguish BentoML’s built-in liveness, readiness, and Prometheus metrics endpoints and explain their use in Kubernetes monitoring.
- Bento package and build configuration. Explain that a Bento is a deployment-ready package containing the trained model, service code, and dependencies. Describe how bentofile.yaml identifies the service file, included files, and base Docker image used to build it.
- Build output and Bento tag. Describe using bentoml build to create the Bento and explain that the build generates a unique tag for that service version in service_name:version_label format.
- Containerized Bento deployment. Explain that the next step is to containerize the built Bento using its tag, rather than push it to BentoCloud in this example. Describe how a container image carrying the service and dependencies improves portability and consistency across deployment environments.
- Explicit Bento version tags. Explain that a Bento tag identifies a particular Bento version and why specifying the full tag is preferable to using only the name.
- Complementary MLflow and BentoML roles. Explain why the section uses MLflow for experiment tracking and model selection, then BentoML for serving the selected model and reducing deployment work.
- Production model loading path. Trace how the recommender service queries MLflow for the URI of the latest registered model with the prod alias, downloads that model locally, and serves it through its API. Explain why a pipeline could pass the URI from a previous step instead of using an MLflow client in the service.
- Local serving and containerization sequence. Describe creating the service file and bentofile, building a Bento, serving it locally, testing /predict with inference requests, and containerizing it once the responses are verified.
- MLflow-only inference workflow. Trace the MLflow-only path from training and logging a model, through registration in the model registry, to serving it as a REST endpoint. Describe local serving and building a Docker model server as deployment options, including the server’s health and inference endpoints.
- Deployment approach tradeoff. Explain why a team might use MLflow alone instead of MLflow with BentoML, and when the combined approach is a better fit.
- KServe deployment workflow. Describe how KServe deploys a model as a Kubernetes microservice: create a namespace, create an InferenceService, and wait for it to initialize and receive a URL when DNS is configured.
- Model serving tool tradeoffs. Explain why a Kubernetes-centric team seeking fine-grained serving control might choose KServe, while a team prioritizing quick, developer-friendly deployment might choose BentoML or MLflow. Distinguish BentoML’s simplicity from MLflow’s broader experiment-tracking and model-versioning platform.
What it builds on
- Validating model artifacts (chapter 8) · Validation of the object detection model artifact.
- Local inference in a notebook (chapter 9) · Local inference with the movie recommender model.
- Continuous integration and deployment (chapter 3) · Introduces continuous integration and deployment for applications.
- Model registry with MLflow (chapter 9) · Introduces registering model versions with MLflow.
- Training the model (chapter 8) · Training the YOLO model used in the serving example.
- Training an object detection model (chapter 8) · The object detection model used in the example.
- Simplifying model training (chapter 9) · Provides the trained YOLOv8 model used for inference.
- Kubernetes objects (chapter 3) · Introduces Kubernetes objects, including the pods managed by health probes.
Section by section
Introduction
The chapter introduces the move from trained, validated models to serving them as APIs, outlines the BentoML deployment path, and explains the value and limits of self-service deployment.
- Path from models to services: Describe how the chapter moves from the trained object detection and movie recommendation models to services: first building and running a BentoML service locally, then packaging it in a container for deployment. Mention that the chapter also considers using BentoML with MLflow and alternatives.
Why it matters: This path connects the models you trained to the services that will make them available for inference. - Self-service deployment tradeoffs: Explain why self-service deployment helps model developers test and iterate with fewer manual errors and bottlenecks, while distinguishing nonproduction testing by data scientists from production deployment by ML engineers.
Why it matters: Self-service lets developers validate deployability early without removing the production handoff.
Key terms: BentoML, model serving, service, container, MLflow, self-service model deployment, early validation, nonproduction environments, production deployment
10.1 Model deployment is hard
This section explains why deploying an ML model adds serving, operational, and governance challenges to an already complex software deployment process.
- ML deployment complexity: Explain how ML deployment builds on conventional software practices such as testing and CI/CD but adds model-specific concerns that require specialized tools and collaboration between data scientists and ML engineers.
Why it matters: A working model is only one part of a reliable production ML system. - Inference and infrastructure choices: Describe how the intended inference pattern—batch predictions, real-time inference, or both—affects deployment, alongside variable workloads, compute-resource allocation, and possible GPU requirements.
Why it matters: A serving setup must fit both how predictions are requested and the resources the model needs. - Model reliability over time: Explain why serving must preserve required input formats and preprocessing, monitor and log model behavior, and manage retraining, new versions, and rollback.
Why it matters: Reliable inference depends on both giving the model suitable inputs and managing how it changes after release. - Production trust requirements: Describe why production deployment may require explanations of model decisions, protection of sensitive information, and defenses against adversarial attacks.
Why it matters: A model can function correctly yet still be unsuitable for production if its decisions or security risks cannot be addressed.
Key terms: ML model deployment, continuous integration (CI), continuous deployment (CD), inference patterns, batch predictions, real-time inference, model scalability, hardware requirements, resource management, data dependencies
10.2 BentoML: Simplifying model deployment
This section explains how BentoML reduces the work of turning developed models into deployable, scalable, and manageable inference services.
- Unified and reproducible serving: Explain how BentoML simplifies deployment by providing a common way to package and serve models from different frameworks. Describe how bundling the model, dependencies, and serving logic makes builds reproducible, while a high-level API reduces serving-code boilerplate.
Why it matters: These capabilities make the path from an experiment to a production service more consistent. - Flexible and efficient inference: Describe BentoML's support for both real-time API serving and batch inference, and explain how Kubernetes integration, adaptive microbatching, and resource controls help serve workloads at scale while balancing throughput and cost.
Why it matters: Different inference workloads need different serving patterns and resource choices. - Production model operations: Explain how BentoML's built-in monitoring and logging, model version control, and MLOps ecosystem integrations support operating a deployed model, including debugging and rolling back versions.
Why it matters: Deployment also requires ways to observe a service and manage changes after release.
Key terms: unified model serving, reproducible builds, API layer abstraction, real-time API serving, batch inference, scalability, adaptive microbatching, resource optimization, monitoring and logging, model management
10.3 A whirlwind tour of BentoML
This section sets up the chapter’s YOLOv8 serving example by introducing BentoML installation, verification, and the project workspace.
- YOLOv8 service deployment target: Describe the plan to deploy the YOLOv8 model trained in the previous chapter as an ML service using a BentoML Service.
Why it matters: This connects the trained model to the serving work that follows. - BentoML installation and workspace: State that the reader should install BentoML, check that it installed correctly, and work in the project's serving directory or an empty project of their own.
Why it matters: These setup steps prepare a workspace for the serving example.
Key terms: YOLOv8, BentoML Service, ML service, BentoML, serving directory
10.3.1 BentoML Service and Runners
The section explains how BentoML API servers and Runners handle inference requests, then outlines two endpoints for a YOLOv8 object detection service.
- Service components and request flow: Trace an inference request through a BentoML Service: an API server receives and validates the HTTP input, then passes arguments to a Runner that wraps the model and performs inference.
Why it matters: This division of work makes the service implementation easier to follow. - Parallel servers and runners: Explain that a Service can have multiple API servers and multiple Runners, with each Runner in its own Python worker. Describe how parallel API instances help handle concurrent load and avoid a single-server bottleneck, while Runner instances can execute in parallel.
Why it matters: Separating and replicating these components helps the service handle changing inference demand. - Object detection endpoint outputs: Distinguish the planned YOLOv8 service endpoints: /inference returns JSON results for an input image, whereas /render returns an image annotated with bounding box, label, and probability. Explain the programmatic versus visual uses of those outputs.
Why it matters: The two outputs make the same model useful to both other systems and people inspecting predictions.
Key terms: BentoML Service, API server, Runner, inference, horizontal scaling, load balancing, parallel processing, Python worker, /inference, /render
10.4 Executing a BentoML Service locally
This section introduces the plan to create and run a BentoML Service locally, starting with a Runner and exposing an object detection model through a REST API.
- Local BentoML serving plan: Describe the planned local workflow: start with a Runner, create and run a BentoML Service, and package the object detection model for access through a REST API during testing and development.
Why it matters: This gives you the purpose and starting point for the hands-on serving work that follows.
Key terms: BentoML Service, Runner, object detection model, REST API
10.4.1 Loading a model with BentoML Runner
This section builds a locally served YOLOv8 BentoML Service, extends it with downloadable image results, and explains its built-in observability endpoints.
- YOLOv8 Runnable and Runner: Describe how the service loads the previous chapter’s YOLOv8 weights in a Runnable, exposes an inference method, and creates a named Runner from that Runnable.
Why it matters: The Runnable and Runner connect the trained model to BentoML’s serving workflow. - Asynchronous image inference endpoint: Trace an image request through the BentoML Service API to the Runner and explain why the endpoint awaits the asynchronous inference result.
Why it matters: This explains how a request gets a completed prediction without unnecessarily blocking the server. - Multiple Runner workflows: Explain how a Service can use multiple Runners for preprocessing and concurrent inference with two object detectors.
Why it matters: Multiple Runners let one API request combine processing stages and model outputs. - Downloadable rendered detection image: Describe how the /render endpoint saves an inference image with bounding boxes and labels and returns it for download.
Why it matters: The endpoint makes detections directly visible to an API user instead of returning only structured predictions. - Built-in service observability endpoints: Distinguish BentoML’s built-in liveness, readiness, and Prometheus metrics endpoints and explain their use in Kubernetes monitoring.
Why it matters: These built-in endpoints help Kubernetes manage pods and let operators observe service health without writing the checks themselves.
Key terms: YOLOv8Runnable, bentoml.Runnable, @bentoml.Runnable.method, bentoml.Runner, yolov8_custom.pt, bentoml.Service, @svc.api, Image(), JSON(), async def
10.5 Building Bentos: Packaging your service for deployment
This section shows how to package a BentoML service as a Bento, build it, and containerize it for deployment.
- Bento package and build configuration: Explain that a Bento is a deployment-ready package containing the trained model, service code, and dependencies. Describe how bentofile.yaml identifies the service file, included files, and base Docker image used to build it.
Why it matters: The build configuration determines what travels with the service when it is deployed. - Build output and Bento tag: Describe using bentoml build to create the Bento and explain that the build generates a unique tag for that service version in service_name:version_label format.
Why it matters: The tag lets you refer to the exact Bento version you built when taking the next deployment step. - Containerized Bento deployment: Explain that the next step is to containerize the built Bento using its tag, rather than push it to BentoCloud in this example. Describe how a container image carrying the service and dependencies improves portability and consistency across deployment environments.
Why it matters: Containerization makes the packaged service easier to deploy without environment compatibility conflicts.
Key terms: Bento, bentofile.yaml, service file, base Docker image, bentoml build, Bento tag, service_name:version_label, bentoml containerize, BentoCloud, container image
10.5.1 Bento tags: Versioning and managing your Bentos
This section explains how Bento tags identify deployable versions and how to verify that a containerized YOLOv8 Bento works.
- Explicit Bento version tags: Explain that a Bento tag identifies a particular Bento version and why specifying the full tag is preferable to using only the name.
Why it matters: An explicit tag helps you deploy the intended version and trace it later. - Containerized Bento verification: Describe how to check that the containerized YOLOv8 Bento starts and serves the expected APIs.
Why it matters: A built container is not ready for deployment until its service actually runs and responds as expected. - Container portability benefit: Explain why packaging the Bento and its dependencies in a Docker container helps deployment across environments.
Why it matters: Packaging dependencies with the service makes it easier to move the same service between platforms.
Key terms: Bento tags, yolo_v8:3jghhcfxvwsrnbsb, bentoml build, workers, API page, /invocation, /render, container image, dependencies, consistency
10.6 BentoML and MLflow inference
The section combines MLflow’s model registry with BentoML to turn the recommender model into a locally tested, containerized inference service.
- Complementary MLflow and BentoML roles: Explain why the section uses MLflow for experiment tracking and model selection, then BentoML for serving the selected model and reducing deployment work.
Why it matters: The two tools support different parts of an iterative model-development workflow. - Production model loading path: Trace how the recommender service queries MLflow for the URI of the latest registered model with the prod alias, downloads that model locally, and serves it through its API. Explain why a pipeline could pass the URI from a previous step instead of using an MLflow client in the service.
Why it matters: This identifies how the service picks up the registered production model rather than a local model file. - Local serving and containerization sequence: Describe creating the service file and bentofile, building a Bento, serving it locally, testing /predict with inference requests, and containerizing it once the responses are verified.
Why it matters: The sequence turns a registered model into a tested, deployable model server. - Built-in serving capabilities: Explain which work BentoML avoids compared with writing a custom API server, including monitoring, health and liveliness endpoints, batching or GPU handling, and input/output documentation.
Why it matters: These built-in capabilities explain why the section treats BentoML as more than a packaging command. - Iterative deployment and Yatai: Describe how quickly testing a model locally and containerizing a modified service supports staging experiments or A/B tests, and state that built Bentos can be pushed to Yatai for centrally managed serving on Kubernetes.
Why it matters: The workflow supports model iteration beyond a single local test.
Key terms: MLflow, BentoML, model registry, experiment tracking, inference service, prod alias, model URI, MLflow client, bentofile, bentoml build
10.7 Using only MLflow to create an inference service
This section shows how to serve an MLflow-registered model without BentoML and explains when that simpler approach is preferable.
- MLflow-only inference workflow: Trace the MLflow-only path from training and logging a model, through registration in the model registry, to serving it as a REST endpoint. Describe local serving and building a Docker model server as deployment options, including the server’s health and inference endpoints.
Why it matters: The workflow connects a tracked model to an inference service without introducing another tool. - Deployment approach tradeoff: Explain why a team might use MLflow alone instead of MLflow with BentoML, and when the combined approach is a better fit.
Why it matters: Choosing a serving approach depends on whether simplicity or deployment flexibility matters more.
Key terms: MLflow tracking APIs, MLflow model registry, REST endpoint, Docker, /health, /invocations, MLflow ecosystem, BentoML, fine-grained deployment options, monitoring
10.8 KServe: An alternative to BentoML
This section introduces KServe’s Kubernetes-based deployment workflow and compares when to choose KServe, BentoML, or MLflow for model serving.
- KServe deployment workflow: Describe how KServe deploys a model as a Kubernetes microservice: create a namespace, create an InferenceService, and wait for it to initialize and receive a URL when DNS is configured.
Why it matters: The workflow shows what Kubernetes setup is involved before a KServe model can be consumed. - Model serving tool tradeoffs: Explain why a Kubernetes-centric team seeking fine-grained serving control might choose KServe, while a team prioritizing quick, developer-friendly deployment might choose BentoML or MLflow. Distinguish BentoML’s simplicity from MLflow’s broader experiment-tracking and model-versioning platform.
Why it matters: The best serving tool depends on a team’s Kubernetes expertise, infrastructure, and project requirements.
Key terms: KServe, K8s namespace, InferenceService, custom resource definition, DNS, BentoML, MLflow, Kubeflow, fine-grained control, model versioning
10.9 Summary
The chapter summary recaps BentoML’s serving workflow and observability features, then compares serving with BentoML and MLflow against simpler or alternative tool choices.
Where this is used later
- Monitoring (chapter 11) · Develops monitoring and logging of deployed systems.
- Data drift detection (chapter 11) · Develops detection of data drift.
- Explainability (chapter 11) · Develops explanations for model outputs.
- Basic monitoring (chapter 11) · Develops basic monitoring of deployed services.
Concepts explained in this chapter
- BentoML serviceA BentoML service is an application object that defines model-serving logic and exposes it through API endpoints.
- BentoML runnerA BentoML runner is a unit of computation that executes on Python worker processes and can be scaled independently within a service.
- health checkA diagnostic test that determines whether a service or execution unit is functioning and able to handle requests.
- async/awaitAsync/await is a programming-language feature for expressing asynchronous operations in a sequential-looking form.
- bentoml runnableA BentoML base class for defining runnable units of model or application computation.
- liveness checkA periodic check that determines whether a running workload should be restarted because its process is no longer healthy.
- multi-runner serviceA service architecture that composes multiple independently executable BentoML runners into one workflow.
- readiness checkA periodic check that determines whether a workload is ready to receive traffic.
- BentoMLBentoML is an open-source framework for packaging and serving machine learning models as production APIs.
- Model ServingA model serving framework is software that packages, exposes, operates, and scales machine learning models for inference.
- MLflowMLflow is an open-source platform for managing the machine learning lifecycle, including experiment tracking, model packaging, registry, and deployment.
- bentoA Bento is a unified package and file format that bundles a machine-learning application, its model, serving logic, dependencies, and configuration.
- deploymentThe process of releasing software into an environment where it can execute and serve its intended users or workloads.
- bentoml buildbentoml build is a BentoML command that builds a Bento from its application definition and dependencies.
- monitoringLogging is the recording of timestamped events, states, and diagnostic information produced by a software system.
- scalabilityThe ability of a system to maintain suitable operation as workload volume or resource demand increases.
- containerA software container is an isolated, portable execution unit that packages an application with its dependencies.
- machine learning systemA software system that incorporates machine learning models and the data, computation, and operational components required to deliver their outputs.
- versioningThe systematic recording and management of changes to files, code, or other managed artifacts over time.
- endpointAn API endpoint is a network-accessible interface that receives requests and returns responses according to an API contract.
- bentofile.yamlA bentofile.yaml file is a declarative BentoML configuration file specifying a service, included files, dependencies, labels, and environment variables for building a Bento.
- observabilityObservability is the ability to infer the internal state and behavior of a deployed system from its outputs and telemetry, including application-specific information flows.
- provenanceData lineage is the record of an artifact's origins, transformations, dependencies, and movement through a data or machine-learning workflow.
- experimentationA/B testing is a controlled comparison of two or more variants by assigning them to different users or inputs and measuring their outcomes.
- batch inferenceBatch inference is the process of applying a trained machine-learning model to a finite collection of inputs in grouped execution rather than one request at a time.
- rollbackA rollback is the restoration of a deployed application to a previously functioning version.
- interpretabilityModel interpretability is the degree to which a model's behavior and predictions can be understood by people or analysis methods.
- driftData drift is a change over time in the statistical distribution or characteristics of input data relative to the data used to develop a model.
- language modelA large language model is a neural language model trained on extensive text data to generate or transform natural-language content.
- privacyThe protection of information about individuals or sensitive entities from inappropriate disclosure or use.
- retrainingModel retraining is the process of training a machine-learning model again using newly available or updated data.
- securitySecurity is the protection of software, models, data, and infrastructure from unauthorized access, misuse, compromise, or attack.
- batchingMicrobatching is the processing of a small group of newly arrived inputs together for analysis or computation.
- experiment trackingAn experiment tracker is a software component that records machine-learning experiment configurations, results, and artifacts.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.