A multi-runner service composes several BentoML Runners behind one Service, enabling multistage inference workflows instead of placing every operation in one model wrapper. section 10.4.1
A workflow can combine runners in different ways:
- Chain preprocessing into inference, such as passing an image through a grayscale converter before an object detector. section 10.4.1
- Fan one prepared input into multiple object detectors and await their results together with asyncio.gather. section 10.4.1
- Expose the composed workflow through a typed API endpoint, using async_run for runner calls and await for completion. section 10.4.1
Parallel detector service
Register preprocessing and detector runners with bentoml.Service, await the preprocessing result, then run object_detector_1 and object_detector_2 concurrently and return named results. section 10.4.1
@svc.api(input=Image(), output=JSON())
async def predict(input_image: PIL.Image.Image) -> str:
model_input = await grayscale_converter.async_run(input_image)
results = await asyncio.gather(
object_detector_1.async_run(model_input),
object_detector_2.async_run(model_input),
)
return {"results": { "model_a": results[0], "model_b": results[1]
Shows the endpoint implementation for preprocessing one image, running two detectors concurrently, and returning model-specific results. section 10.4.1
