A BentoML service is the application object that connects an inference API to its serving logic. It wraps API servers and one or more Runners: an API server receives, parses, and validates requests, while a Runner performs model computation. Multiple API-server and Runner instances can handle concurrent requests and increase throughput. section 10.3.1
Build a service from the model outward:
- Prepare the model. For an existing MLflow model, import a selected model stage into the local BentoML store. Alternatively, load model weights in a custom bentoml.Runnable; direct loading keeps a first service simple, while registry-based management is deferred for versioning and updates. section 6.1.1 section 10.4.1
- Create a Runner, then pass it to bentoml.Service(..., runners=[...]). A service can hold multiple Runners for processing stages or parallel model calls. section 10.4.1
- Decorate endpoint functions with @svc.api and declare input and output types. Endpoint code prepares inputs, calls a Runner, and returns the response; async endpoints can await Runner calls. section 10.4.1 section 6.1.1
- For a distributable application, point bentofile.yaml at the service object and include its code, requirements, and environment configuration. Build the resulting Bento for deployment. section 6.1.1 section 10.9
import bentoml
yolo_v8_runner = bentoml.Runner(
YOLOv8Runnable, name="yolov8_runnable") #1
svc = bentoml.Service(
"yolo_v8",
runners=[yolo_v8_runner]) #2
The minimal object-detection wiring: construct a Runner from a custom Runnable, then register that Runner on the service. section 10.4.1
Prediction endpoint with online features
An income-classification service exposes /predict. Its endpoint accepts a validated user input, retrieves online features from Feast, maps them into the model’s expected columns, invokes the classifier Runner, and returns an income category with the user ID. Startup initialization prepares shared configuration and state used by the endpoint. section 6.1.1
Listing 6.3 Defining the service endpoint
@svc.api(input=full_input_spec, output=JSON(), route="/predict")
def predict(
inputs: IncomeClassifierUsers,
ctx: bentoml.Context
) -> Dict[str, Any]:
input_dict = inputs.dict() #1
feature_df = (
ctx.state["store"]
.get_online_features(
features=ctx.state["feature_list"],
entity_rows=[input_dict],
)
.to_df()
) #2
feature_df.drop(columns=["user_id"], inplace=True)
data_mapper = InputMapper(feature_df, ctx.state["col_list"])
input_df = data_mapper.generate_pandas_dataframe() #3
output_mapper = OutputMapper(
income_clf_runner.predict.run(
input_df
)[0]
) #4
return {
"income_category": output_mapper.map_prediction(),
"user_id": input_dict["user_id"],
} #5
The /predict listing shows where feature retrieval and data mapping sit between the HTTP input and the Runner call. section 6.1.1
For an image service, Image() and JSON() define an inference endpoint’s boundary; a separate render endpoint can return an annotated image. Local execution supports API testing before deployment. In Kubernetes environments, health checks, readiness and liveness probes, and Prometheus metrics support operation and monitoring. section 10.3.1 section 10.4 section 10.9
