Model serving packages a trained model with its inference logic and exposes it as a dependable service. It turns a validated artifact into an API that applications can call, while addressing deployment repeatability, health checks, scaling, monitoring, and resource management. section 10.0 section 10.2
BentoML provides a practical implementation path: define a service, configure resources and request timeouts, package dependencies in a Bento, run the service locally, test its inference endpoint, and containerize the result. It abstracts custom API, health, liveness, batching, and device-handling work while retaining deployment controls. section 10.6 section 10.2
A production serving design commonly includes:
- A model source such as an MLflow registry alias, allowing the service to retrieve the current production model rather than embedding a fixed artifact. section 10.6
- A reproducible package containing service code, dependencies, labels, and configuration, followed by local validation before deployment. section 10.6
- A deployment target ranging from local serving to Docker and Kubernetes, with monitoring, logging, version management, rollback, and scaling handled according to operational needs. section 10.7 section 10.2
BentoML with MLflow
A recommender service queries MLflow for the model version assigned the prod alias, imports that model into BentoML, builds a Bento, serves it locally, tests /predict, and containerizes the validated service. MLflow alone can provide a simpler alternative through /health and /invocations, while BentoML is preferable when fine-grained deployment options and monitoring are required. section 10.6 section 10.7
Listing 10.1 Using MLflow and BentoML in an inference service
import bentoml
import mlflow
import torch
import numpy as np
from mlflow import MlflowClient
@bentoml.service(
resources={"cpu": "2"},
traffic={"timeout": 10},
)
class RecommenderRunnable:
def __init__(
self,
registered_model_name = 'recommender_production',
device = 'cpu'):
mlflow.set_tracking_uri(uri="http://mlflow:8080") #1
client = MlflowClient() #1
current_prod = client.get_model_version_by_alias(
registered_model_name,
"prod") #1
model_uri = f"runs:/{current_prod.run_id}/model" #1
bentoml.mlflow.import_model(
"recommender", model_uri) #1
bento_model = bentoml.mlflow.get(
"recommender:latest") #1
mlflow_model_path = bento_model.path_of(
bentoml.mlflow.MLFLOW_MODEL_FOLDER)
self.model = mlflow.pytorch.load_model(mlflow_model_path)
self.device = device
self.model.to(self.device)
self.model.eval()
@bentoml.api #2
def predict(
self,
user_id: int,
top_k: int=10,
ranked_movies:np.ndarray=None) -> np.ndarray:
user_id = torch.tensor(
[user_id],
dtype=torch.long).to(self.device)
all_items = torch.arange(
1,
self.model.n_items + 1,
dtype=torch.long).to(self.device)
if ranked_movies is not None: #1
ranked_movies = torch.tensor(
ranked_movies,
dtype=torch.long).to(self.device)
unrated_items = all_items[
~torch.isin(all_items, ranked_movies)]
else:
unrated_items = all_items
user_ids = user_id.repeat(len(unrated_items))
with torch.no_grad():
predictions = self.model(
user_ids,
unrated_items).squeeze() #3
top_n_indices = torch.topk(
predictions, top_k).indices #4
recommended_items = unrated_items[top_n_indices].cpu().numpy()
return recommended_items
Inspect the service listing to see how resource limits, timeouts, MLflow model selection, and BentoML model loading fit together. section 10.6
