A BentoML runner is the computation unit behind a serving API. The API server parses and validates a request, then passes its inputs to a runner that executes model inference. Each runner runs in its own Python worker, allowing multiple runner instances to execute in parallel. A service can contain more than one runner. section 10.3.1
To put a runner into a service:
- Create it from a stored model with `.to_runner()`, or define a `bentoml.Runnable` whose methods perform the computation and pass that class to `bentoml.Runner`. section 6.1.1 section 10.4.1
- Register the runner in `bentoml.Service(..., runners=[...])`, then call its method from an API endpoint. Use `run(...)` for the synchronous prediction shown here, or await `async_run(...)` from an asynchronous endpoint. section 6.1.1 section 10.4.1
Income classification
An MLflow model is imported into the local BentoML store and converted to a runner. The `/predict` endpoint retrieves online features from Feast, maps them into a dataframe, calls `income_clf_runner.predict.run(input_df)`, and maps the prediction into the response. Feature retrieval and response formatting stay in the service endpoint; inference runs through the runner. section 6.1.1
import bentoml
import json
from ultralytics import YOLO
class YOLOv8Runnable(bentoml.Runnable):
def __init__(self):
self.model = YOLO("yolov8_custom.pt")
@bentoml.Runnable.method(batchable=False) #1
def inference(self, input_img): #2
results = self.model(input_img)[0] #3
return json.loads(results[0].tojson()) #4
The custom `YOLOv8Runnable` loads a model in its constructor and marks `inference` as a runner-callable method. section 10.4.1
@svc.api(input=Image(), output=JSON())
async def predict(input_image: PIL.Image.Image) -> str:
model_input = await grayscale_converter.async_run(input_image)
results = await asyncio.gather(
object_detector_1.async_run(model_input),
object_detector_2.async_run(model_input),
)
return {"results": { "model_a": results[0], "model_b": results[1]
Multiple runners can form a workflow: one converts an image, then two detectors run concurrently on the converted input. section 10.4.1
Keep the distinction operational: the service owns the API, while runners own the computation. Register the runners an endpoint needs, and await asynchronous calls before using their results. section 10.3.1 section 10.4.1
