Chapter 6: Productionizing ML models

Introduction

The chapter introduces production deployment with BentoML and data drift monitoring with Evidently, using the income classification model as its running example.

• BentoML model deployment roadmap: Explain that after orchestrating ML pipelines, the next challenge is reliably serving models as APIs in production, and that BentoML simplifies deployment so engineers can focus on serving logic rather than infrastructure setup.
  Why it matters: This connects the trained model to the production service that will use it.
• Evidently data drift monitoring: Explain that changing real-world data distributions can degrade a deployed model’s performance, and that Evidently is introduced to detect and analyze that drift so teams can maintain performance proactively.
  Why it matters: Deployment is not the end of the ML life cycle; changing data can make a working model less effective.
• Chapter examples and LLM scope: Describe the income classification use case as the chapter’s practical example, and distinguish this chapter’s broadly applicable deployment and monitoring techniques from the dedicated production LLM guidance reserved for chapters 12 and 13.
  Why it matters: This sets expectations for what the chapter demonstrates and what the later LLM chapters address.

Key terms: BentoML, deployment manager, APIs, serving logic, data drift, Evidently, data distributions, monitoring, income classification, large language models (LLMs)

6.1 BentoML as a deployment platform

This section introduces BentoML as a way to turn the previously batch-deployed income classifier into an API service by packaging it as a Bento and deploying it through Yatai.

• BentoML deployment workflow: Trace the two main steps for deploying the income classifier as an API endpoint: build a Bento locally, then push it to Yatai for containerization and deployment on Kubernetes.
  Why it matters: This workflow connects the packaged model application to a running API service.
• Packaging and operations roles: Distinguish a Bento, the file archive that packages an ML application and its requirements, from Yatai, which deploys, operates, scales, and monitors the service on Kubernetes.
  Why it matters: Keeping the package and deployment component separate makes the BentoML process easier to explain.
• Deployment simplification rationale: Explain why the section uses BentoML instead of repeating the earlier manual API deployment steps: building a Docker image, configuring Kubernetes manifests, adding monitoring logic, and setting up CI/CD take time and require skills not every data scientist has.
  Why it matters: The earlier deployment work explains why a streamlined ML deployment platform is useful.

Key terms: BentoML, Bento, Yatai, API endpoint, unified distribution format, Kubernetes, Docker image, Kubernetes manifests, monitoring logic, CI

6.1.1 Building a Bento

This section builds an income-classifier Bento by importing an MLflow model, defining a BentoML inference service, and specifying how to package the application.

• MLflow model import: Explain how the registered MLflow model is brought into the local BentoML model store and how to check that the import worked.
  Why it matters: The service needs a locally available model before it can package and serve predictions.
• Service runner and startup state: Describe how the BentoML service and prediction runner are created, and what the service initializes at startup for feature retrieval.
  Why it matters: Separating serving logic from independently scalable inference workers makes the service practical to run.
• Income prediction API flow: Trace how a user_id request becomes an income-category response through Feast feature retrieval and model prediction.
  Why it matters: The endpoint connects an incoming entity ID to the features and prediction the deployed application actually returns.
• Bento packaging configuration: Describe what bentofile.yaml specifies to package the service, its files, dependencies, and environment configuration.
  Why it matters: The Bento needs the serving code and its runtime configuration bundled before it can be built and deployed.

Key terms: bentoml.mlflow.import_model, MLflow model URI, BentoML local store, bentoml models list, bentoml.Service, runner, bentoml.mlflow.get(model_name).to_runner(), @svc.on_startup, context.state, Feast

6.1.2 Building and pushing the Bento

This section builds an income-classifier Bento, tests it locally, and pushes it to Yatai for Kubernetes deployment.

• Bento build and inspection: Describe how the defined service and bentofile.yaml are turned into a Bento with bentoml build, then inspected with bentoml list in the local Bento Store. Distinguish the listed application Size from Model Size.
  Why it matters: Building and identifying the Bento is the first step toward deploying the service.
• Local Bento endpoint testing: Describe testing the built Bento locally, including running its Docker container, port-forwarding the MinIO, MLflow, and Feast Redis Online Store services, and supplying a user_id to the /predict endpoint at localhost:3000 after downloads finish.
  Why it matters: A local prediction test checks the packaged service before it is pushed for deployment.
• Remote Bento Store push: Explain why and how the tested Bento is pushed to the remote Bento Store: port-forward to Yatai, log in with an API token, and use bentoml push to send the latest income-classifier Bento, including its model and application, for subsequent Kubernetes deployment.
  Why it matters: Yatai needs the remote Bento before it can pull and deploy the application.

Key terms: bentoml build, bentoml list, local Bento Store, Size, Model Size, docker run, minio-service, mlflow-service, Feast Redis Online Store, user_id

6.1.3 Deploying a Bento

This section walks through deploying a pushed income-classifier Bento with Yatai, checking its API and metrics, and explaining what BentoML handles automatically.

• Yatai deployment workflow: Describe how to verify that the model and Bento are in Yatai, then create a deployment by selecting the Bento and its tag, setting deployment options, and submitting it.
  Why it matters: These are the steps that turn the pushed Bento into a running application.
• Image build and pod roles: Explain that Yatai builds a Docker image through a Kubernetes job and launches API Server and Runner pods, with the API Server receiving requests and the Runner holding models and performing inference.
  Why it matters: The two pod roles explain how the deployed service handles predictions and how its parts can scale.
• Deployed API verification: Describe how to find the generated ingress, open the service’s Swagger document, and test the /predict endpoint with a suitable user ID.
  Why it matters: A live deployment still needs an accessible endpoint and a successful prediction check.
• Built-in metrics and deployment automation: Explain that BentoML supplies a /metrics endpoint for Prometheus and handles image building, Kubernetes deployment setup, and metrics wiring without hand-writing a Dockerfile, manifests, or metrics logic, given a bentofile.yaml.
  Why it matters: This is why the section presents BentoML as a way to standardize ML deployments.

Key terms: Yatai, Models tab, Bentos tab, Deployments tab, Bento tag, replicas, Kubernetes job, Docker image, API Server, Runner

6.2 Evidently for data drift monitoring

The section explains why deployed models need data drift monitoring and introduces how Evidently compares historical and inference data to detect changes.

• Production data drift risk: Explain how a model that initially performs well can lose accuracy when production data changes from the historical data on which it was trained, and why that change calls for detection and corrective action.
  Why it matters: A deployed model can keep running even when the data it sees no longer supports reliable predictions.
• Forms of data drift: Distinguish changes in the target label, class proportions, input-feature distributions, and abrupt changes caused by external events using the section's examples.
  Why it matters: Naming what changed helps clarify the drift a team needs to investigate.
• Evidently drift detection tests: Describe how Evidently uses statistical tests to compare historical and inference data, what the KS test, chi-square test, and Wasserstein distance indicate, and how a user can override Evidently's test selection.
  Why it matters: These comparisons turn a suspected change in production data into evidence that can be examined.

Key terms: data drift, historical data, inference data, label drift, prior probability shift, covariate shift, sudden drift, Evidently, Kolmogorov–Smirnov Test, KS Test

6.2.1 Data drift detection report and dashboard

This section shows how to generate and interpret Evidently data drift reports, then set up a dashboard to track results across data snapshots.

• Evidently metrics and presets: Distinguish individual Evidently metrics from metric presets: metrics measure dataset- or column-level properties, while presets bundle metrics into reports for uses such as data drift or data quality.
  Why it matters: This distinction explains why a preset can generate a useful report with little code.
• Data drift report workflow: Describe how to generate a data drift report by creating a Report with DataDriftPreset, running it on historical reference and current data frames, and saving it as HTML.
  Why it matters: The reference-versus-current comparison is the basis for the section’s drift checks.
• Column drift result interpretation: Explain how a drift report compares each feature’s reference and current distributions and shows the statistical test and its drift score; for the illustrated Wasserstein distance test, a score above Evidently’s default threshold of 0.1 is classified as drift.
  Why it matters: Reading the per-column results lets you tell which features the report flags as drifted.
• Snapshot dashboard setup: Explain why repeated snapshot reports are needed to follow changes over time, and describe creating an Evidently UI workspace and project, adding dashboard panels, and saving the project. Describe how a counter panel can show the latest share of drifted columns using a metric field and LAST aggregation.
  Why it matters: A configured dashboard makes results from successive snapshots easier to monitor than isolated reports.

Key terms: metric, metric preset, DataDriftPreset, DataQualityPreset, RegressionPreset, reference dataset, current dataset, Report, save_html, Reference Distribution

6.2.2 Data drift detection Kubeflow pipeline component

This section adds an Evidently data drift check to the income classifier’s batch inference pipeline and uses its reports to populate a dashboard.

• Batch inference drift gate: Explain where the drift detection component sits in the income classifier’s Kubeflow inference pipeline and what happens when new feature data has drifted from the training reference data.
  Why it matters: Checking for drift before inference lets the pipeline stop when its incoming data differs significantly from the training data.
• Drift component data flow: Trace how the KFP component obtains its reference and current datasets, runs an Evidently report with metrics matching the dashboard, and adds report data to the Evidently workspace.
  Why it matters: The report needs both datasets and the dashboard’s metrics to make the batch drift results visible.
• Dashboard results across runs: Describe what the example dashboard shows after a successful pipeline run and how subsequent runs make drift statistics viewable over time.
  Why it matters: The dashboard turns each batch check into results that can be inspected across runs.

Key terms: data drift, reference data, retrieve_features_task, run inference component, MLflow client, run ID, MinIO, Feast, Evidently workspace, report.run

6.2.3 Data drift detection for a model deployed as an API

This section adds sliding-window data drift monitoring to a BentoML prediction API, shows how to interpret its reports, and outlines what to do when drift appears.

• Microbatch drift comparison rationale: Explain why the API compares a window of recent prediction requests with reference data instead of running a drift test for each request.
  Why it matters: A single request is too small a sample for a reliable drift comparison.
• Sliding window report cycle: Trace how MonitoringService fills a window with incoming data, generates reports once enough data has arrived, and keeps the window current as further requests arrive.
  Why it matters: The moving window produces repeated drift snapshots without comparing the reference data to just one request.
• BentoML monitoring integration: Describe where the monitoring service is initialized and how the prediction endpoint feeds it data, including the reference dataset, workspace, and configuration it needs.
  Why it matters: Drift monitoring works only if the deployed API supplies each request to a configured monitoring service.
• Drift dashboard interpretation: Explain what the repeated dashboard points represent and why the example's high drift reading does not necessarily indicate a real data change.
  Why it matters: A drift alert needs interpretation before it is treated as evidence that the production data changed.
• Drift investigation and remedies: Describe how to assess a detected drift and name possible responses when it affects the application.
  Why it matters: Detecting drift is useful only when the team checks its impact and addresses its cause.

Key terms: reference dataset, microbatch, /predict endpoint, MonitoringService, window size, current data, report snapshots, RemoteWorkspace, MLflow, iterate

6.3 Summary

The chapter summary connects streamlined model deployment with the need to understand its infrastructure and monitor data drift after deployment.