BentoML packages a model, its dependencies, and serving logic into a reproducible Bento, then exposes the model through an inference service. It reduces the deployment work that otherwise requires a Dockerfile, Kubernetes manifests, and custom monitoring code. The practical goal is to move from a trained model to a testable API without making every model developer build that infrastructure by hand. section 6.1 section 10.2
A production-oriented path has four steps:
- Define the inference service and its API endpoints. BentoML supports service APIs and decorators; a service can use a Runner to perform inference. section 10.9
- Declare the service, included files, and Python requirements in a bentofile; run `bentoml build` to produce a versioned Bento. section 10.6 section 10.9
- Serve the Bento locally and test its endpoint before deploying it. For a Kubernetes deployment, push the Bento to Yatai, which builds the container image and launches the service. section 10.6 section 6.1
- Inspect the deployed API and its metrics. Yatai can scale API Server pods, which receive requests, separately from Runner pods, which hold models and perform inference. section 6.1.3
Income-classifier API
An income-band classifier illustrates the difference between inference patterns: Kubeflow Pipelines generates batch predictions, while BentoML serves real-time requests through an API. After deployment, a Swagger document exposes `/predict` for a test request with a user ID. The service also exposes `/metrics` for Prometheus scraping, including request-rate and latency metrics. These operational signals do not replace checks for data drift; Evidently is used for that separate task. section 4.0 section 6.1.3 section 6.0
Listing 10.2 Sample bentofile to create the inference service
service: "service:RecommenderRunnable" labels: owner: mlops stage: demo include: - "*.py" python: requirements_txt: './requirements.txt'
The sample bentofile shows the concrete packaging boundary: a service entry point, labels, included Python files, and a requirements file. section 10.6
BentoML can serve a model selected through MLflow rather than only a local model. In the recommender example, the service queries MLflow for the model version aliased `prod`, imports that model into BentoML, builds a Bento, and runs it locally. MLflow supplies experiment tracking and model lineage; BentoML supplies the serving and deployment experience. Using MLflow alone may be simpler when a smaller organization values fewer tools over the additional serving features. section 10.6 section 10.9
BentoML standardizes deployment, but operating the service still calls for attention to scalability, resource allocation, and monitoring. Its health checks, readiness and liveness probes, and Prometheus metrics support Kubernetes operation; understanding Docker and Kubernetes remains useful when a deployment needs customization. Self-service deployment reduces handoffs without eliminating the role of production deployment engineers. section 10.9 section 6.3 section 10.0
