Chapter 11: Monitoring and explainability

Introduction

This chapter introduces operational monitoring, data drift detection, and explainability as ways to keep deployed ML services reliable and understand their decisions.

• Operational service monitoring: Explain that basic monitoring checks whether a deployed model service meets performance and reliability requirements, including uptime, throughput, latency, and response quality. Describe the chapter’s plan to use BentoML dashboards and custom metrics, logs, and alerts to spot and respond to incidents.
  Why it matters: A model must remain a dependable service after deployment.
• Production data drift monitoring: Explain that data drift monitoring checks whether incoming data and its relationship to the target remain consistent with training. Describe how identifying drift can guide changes to features, training data, or the model, and name deepchecks as the planned monitoring tool.
  Why it matters: A healthy service can still produce poorer predictions when production data changes.
• Model decision explainability: Explain that explainability helps identify which features or inputs contribute to model behavior and changes in that behavior. Describe its role in diagnosing problems and communicating model decisions to stakeholders.
  Why it matters: Understanding why a model behaves as it does supports troubleshooting, trust, and accountability.

Key terms: basic monitoring, service level agreements (SLAs), error budget, BentoML, custom metrics, Alertmanager, data drift monitoring, incoming data, target variable, deepchecks

11.1 Monitoring

This section explains why production applications need monitoring and alerting, then introduces their use in the object detection and movie recommender projects.

• Production monitoring purpose: Explain how monitoring helps keep an application production-ready by detecting anomalies, faults, and failures early, so teams can reduce disruptions and maintain stability and SLAs.
  Why it matters: Monitoring gives teams a chance to address problems before they become prolonged service disruptions.
• Metrics and actionable alerts: Describe how an effective monitoring solution watches performance and business metrics, including resource utilization and response times, and alerts people who can respond to critical incidents or abnormal behavior.
  Why it matters: Metrics reveal trouble, while alerts bring it to the attention of people who can act quickly.

Key terms: monitoring, performance anomalies, SLAs, performance metrics, business metrics, resource utilization, response times, alerting

11.1.1 Basic monitoring

This section shows how to verify that Prometheus collects BentoML deployment metrics and import a prebuilt Grafana dashboard for basic monitoring.

• Basic monitoring metric categories: Explain why a deployed API needs both resource-utilization metrics and request-tracking metrics, including what problems each can reveal.
  Why it matters: Watching both capacity and request behavior helps you spot performance problems before they affect users.
• Prometheus scrape verification: Describe how to check that Prometheus discovers and scrapes the metrics exposed by BentoML deployments, and what to do if the expected PodMonitor is missing.
  Why it matters: Grafana cannot display deployment metrics unless Prometheus is collecting them.
• Prebuilt Grafana dashboard setup: Describe how to import BentoML’s prebuilt dashboard into Grafana and identify the basic monitoring information it displays.
  Why it matters: The prebuilt dashboard makes the collected metrics usable without writing PromQL queries and visualizations from scratch.
• Custom metric motivation: Explain why the default BentoML dashboard may need custom application metrics, using the object detection example.
  Why it matters: Service health metrics do not necessarily show the application-specific behavior needed for reporting and alerts.

Key terms: resource utilization, request tracking metrics, CPU, memory, response time latency, non-200 status codes, /metrics, Prometheus, Service Discovery, PodMonitor

11.1.2 Custom metrics

This section adds application-specific Prometheus metrics to two ML services and shows how to inspect and visualize them in Grafana.

• Object detection confidence histogram: Explain why the object detection service tracks prediction confidence and how it records scores in a Prometheus Histogram: define buckets from 0 to 1 and call observe with each confidence score during inference.
  Why it matters: A confidence distribution can reveal periods of low-confidence predictions that may indicate poor-quality images or degrading model performance.
• Confidence metric visualization: Describe how to verify that the confidence histogram is exposed for Prometheus scraping and how to visualize its 90th-percentile confidence over the last five minutes using PromQL and a Grafana Gauge.
  Why it matters: Exposing and plotting the metric turns recorded predictions into a view of recent model behavior.
• Ranked movie request counter: Explain how the movie recommender uses a Prometheus Counter to track requests that include ranked movies: increment it when ranked_movies is not None, then plot it as a Grafana time series. Distinguish its cumulative behavior from the confidence Histogram.
  Why it matters: A Counter tracks how often a particular request condition occurs, complementing the distribution of confidence scores.

Key terms: confidence score, Histogram, buckets, observe, /invocation, /metrics, le, PromQL, 90th percentile, Gauge

11.1.3 Logging

This section shows how BentoML logs complement metrics and how Loki brings logs into Grafana for centralized monitoring.

• Logs and metrics together: Explain that metrics show measurable behavior such as CPU usage and request counts, while logs provide error messages, stack traces, and event details that help trace failures and find their causes.
  Why it matters: Using both helps you see that something is wrong and investigate why.
• BentoML service logging: Describe how to set up the BentoML logger in service.py using Python's logging library, a format, the BentoML logger, and a log level; give the example of logging an object-detection processing error that leaves the result empty.
  Why it matters: Useful service logs make production errors easier to trace and debug.
• Centralized logs with Loki: Justify bringing logs and metrics together in Grafana rather than leaving metrics there and viewing logs separately in GCP Stackdriver or AWS CloudWatch; explain that Loki aggregates logs and indexes metadata rather than full log contents.
  Why it matters: A shared view makes incidents easier to investigate across sources and teams.
• Grafana log exploration: Describe installing Loki with Helm, then selecting Loki as the Grafana data source, choosing a BentoML deployment and time range, optionally filtering further, and searching its logs for text.
  Why it matters: These steps turn collected logs into something you can inspect during an incident.

Key terms: metrics, logs, logging, observability, BentoML logger, Python logger, service.py, log level, Loki, Grafana

11.1.4 Alerting

This section sets up Prometheus alerts for BentoML service availability and uses Alertmanager to deliver notifications when those alerts fire.

• Prometheus and Alertmanager roles: Explain why uptime alerts matter and distinguish Prometheus generating alerts from Alertmanager routing notifications to recipients.
  Why it matters: An alert helps protect service availability only if a problem is detected and the right people hear about it.
• Unavailable versus missing metrics: Distinguish using Prometheus’s up metric to detect a target that cannot be scraped from using absent to detect a metric that disappears when a deployment is terminated or deleted.
  Why it matters: A terminated deployment may leave no metric for an up-based alert to evaluate.
• Alert rules and firing states: Describe adding PromQL-based BentoML alert rules to the Prometheus Helm values file, including a sustained-condition delay, message, and labels; trace an alert from inactive through pending to triggered.
  Why it matters: Rules and their timing determine when an availability problem becomes an actionable alert.
• Alertmanager notification routing: Describe configuring Alertmanager in Helm values to send fired alerts to Gmail using an app password, and explain how routing rules can direct alerts to different receivers by label or namespace.
  Why it matters: Routing gets each alert to an appropriate recipient rather than merely displaying it in Prometheus.

Key terms: Prometheus, Alertmanager, alerting, uptime, up metric, absent function, MissingUpMetric, PromQL, alert group, serverFiles

11.2 Data drift detection

This section introduces an extension of the book’s data drift detection work from a tabular income classifier to object detection and movie recommendation projects.

• Data Drift Project Extension: Describe how this section builds on chapter 6’s use of statistical tests to detect drift in tabular income-classifier data, in both real-time and batch use cases, and turns to object detection and movie recommendation projects.
  Why it matters: This connects the earlier drift-detection work to the two projects discussed next.

Key terms: data drift, statistical tests, real-time, batch, object detection, movie recommendation

11.2.1 Object detection

This section shows how to detect image-property drift in an object detection project with deepchecks, then describes how to monitor that drift in production.

• Image property drift comparison: Explain why object detection drift is assessed by comparing distributions of image properties, such as brightness or aspect ratio, between training and new images rather than relying on predefined tabular features.
  Why it matters: Image inputs need properties that can be measured and compared before their drift can be monitored.
• Brightness shifted test dataset: Describe how the example deliberately creates drift by changing the brightness of test ID card images while retaining unmodified training images, and how the two image-and-label sets are prepared for deepchecks.
  Why it matters: A controlled brightness change makes it possible to see whether the drift check detects the intended difference.
• Deepchecks drift results: Trace how deepchecks ImagePropertyDrift compares the train and test VisionData, and explain why the example reports brightness drift but no area or aspect-ratio drift.
  Why it matters: Interpreting drift by property helps distinguish the deliberate change from properties that stayed stable.
• Production image drift monitoring: Describe the proposed production workflow: retain images processed by the object detection BentoML Service with their predicted labels in a bucket or database, then periodically compare their image properties with those of the training images for significant drift.
  Why it matters: The example becomes useful in production when incoming images are retained and checked against the training baseline over time.

Key terms: data drift, image properties, brightness, contrast, aspect ratio, training images, ImageEnhance, PIL, brightness_factor, IDCardDataset

11.2.2 Movie recommender

This section demonstrates how to detect ratings drift in a movie recommender by comparing latent-factor distributions from models trained on drifted and unchanged data.

• Latent factors as drift signals: Explain why a deployed matrix-factorization recommender should monitor user and item factors as well as retrain for new users or items. Describe how shifts in those factors can signal changing user preferences or item popularity and help identify risks to production performance.
  Why it matters: Monitoring factor distributions can reveal changes in the ratings data that matter to a deployed recommender.
• Movie rating drift experiment: Trace how the section creates drift by raising selected movie ratings by one without exceeding five, trains the model on drifted and nondrifted datasets, and compares extracted factors with deepchecks FeatureDrift. Describe the data preparation for the check and what its result showed for one movie.
  Why it matters: The experiment connects an intentional change in ratings to a measurable change in the model’s learned factors.

Key terms: matrix factorization, user-item rating matrix, user factors, item factors, data drift, latent factors, MovieLens, item ratings, user and item factors, embeddings

11.3 Explainability

This section explains why model decisions need to be interpretable, distinguishes two approaches to explainability, and shows why explanations matter for object detection and movie recommendations.

• Purpose of model explainability: Explain how explainability differs from monitoring and why understanding individual predictions helps with debugging, improvement, trust, fairness, and regulatory requirements. Describe how a lender's explanation of a credit denial illustrates its practical importance.
  Why it matters: Understanding why a model made a decision helps teams improve it and helps affected people assess that decision.
• Two approaches to explainability: Distinguish model-based explainability, which uses an inherently understandable model structure, from post hoc explainability, which analyzes a trained opaque model to explain its predictions.
  Why it matters: The distinction clarifies whether an explanation comes from the model's structure or from a method applied afterward.
• Project-specific reasons for explanations: Describe what explanations can reveal in object detection and why explanations of movie recommendations can help users and businesses.
  Why it matters: The value of an explanation depends on the decision the model makes and who uses it.

Key terms: explainability, interpretable AI, explainable AI (XAI), monitoring, credit risk models, Equal Credit Opportunity Act (ECOA), model-based explainability, post hoc explainability, linear regression, decision trees

11.3.1 Object detection

This section shows how to generate and interpret Eigen-CAM heatmaps to check what an object detection model focuses on when detecting ID cards.

• Eigen-CAM visual explanations: Explain that Eigen-CAM overlays a heatmap on an image to show which regions most influence an object detection model’s prediction, without modifying or retraining the model.
  Why it matters: A heatmap makes an otherwise opaque prediction easier to inspect.
• Object detection heatmap workflow: Describe the section’s workflow: load the object detection model, identify target layers, initialize Eigen-CAM for object detection, then supply images and plot their heatmaps. Explain that the target layers provide abstract features and spatial information useful for predicting boxes and classes.
  Why it matters: These steps connect the explanation technique to the model and images being inspected.
• ID-card focus checks: Describe how the ID-card heatmap is interpreted and used: the model focuses on the ID card as intended but also attends to image corners. Explain how checking training or inference images periodically, including after retraining, can reveal unwanted focus and guide model changes.
  Why it matters: Checking where a detector focuses can expose reliance on irrelevant image features even when it detects the intended object.

Key terms: Eigen-CAM, heatmap, Class Activation Mapping (CAM), target layers, object detection, bounding boxes, class probabilities, ID card, training, inference

11.3.2 Movie recommendation

This section shows how an explainable matrix factorization model produces movie recommendations with explanations based on similar users’ ratings.

• EMF Neighborhood Explainability: Explain how the EMF model uses similar users or items in latent space to make recommendations explainable. Describe how its explainability score weights training so that explainable user–item pairs have nearby latent representations.
  Why it matters: This connects the model’s training to the explanations it can provide.
• Movie Explainer Setup: Describe the demonstration workflow: fit an EMFModel on a subset of the MovieLens 100K training data, wrap it in a Recommender, and build an EMFExplainer to explain the recommendations.
  Why it matters: The workflow shows how the model is turned into recommendations with explanations.
• Recommendation Explanation Interpretation: Interpret the explanations attached to each user’s top ten recommendations as counts of similar users by rating. Explain why an explanation may be empty and how the counts can support confidence in a prediction.
  Why it matters: These counts make a recommendation understandable to users and the business without promising that every recommendation has an explanation.

Key terms: explainable matrix factorization, EMF, latent space, explainability score, user and item neighborhood, MovieLens 100K Dataset, EMFModel, Recommender, EMFExplainer, recs_with_explanations

11.4 Summary

The chapter summary recaps how monitoring, drift detection, and explainability help keep production ML systems reliable, accurate, and understandable.