gRPC moves inference requests across process or network boundaries using compact binary Protocol Buffers instead of JSON. For large prompts and other ML payloads, sending more information in fewer bytes can substantially reduce network latency and improve throughput. section 6.3.3
Use gRPC when network transmission is the bottleneck, but weigh its setup cost:
- Prefer it for inference services carrying large payloads, especially when latency and throughput matter. section 6.3.3
- Expect more client complexity than a typical curl request because messages must be serialized and deserialized. section 6.3.3
- Use grpcio and define protobuf messages in .proto files when implementing an API without an existing inference service; the learning curve is steeper than FastAPI-based REST. section 6.3.3
- Check whether the inference service already supports gRPC alongside REST; widely implemented protocols can remove much of the setup burden. section 6.3.3
Listing 6.13 Example client using gRPC
import json
import grpc
from mlserver.codecs.string import StringRequestCodec
import mlserver.grpc.converters as converters
import mlserver.grpc.dataplane_pb2_grpc as dataplane
import mlserver.types as types
model_name = "grpc_model"
inputs = {"message": "I'm using gRPC!"}
inputs_bytes = json.dumps(inputs).encode("UTF-8") #1
inference_request = types.InferenceRequest(
inputs=[
types.RequestInput(
name="request",
shape=[len(inputs_bytes)],
datatype="BYTES",
data=[inputs_bytes],
parameters=types.Parameters(content_type="str"),
)
]
)
serialized_request = converters.ModelInferRequestConverter.from_types( #2
inference_request, model_name=model_name, model_version=None
)
grpc_channel = grpc.insecure_channel("localhost:8081") #3
grpc_stub = dataplane.GRPCInferenceServiceStub(grpc_channel)
response = grpc_stub.ModelInfer(serialized_request)
print(response)
deserialized_response = converters.ModelInferResponseConverter.to_types( #4
response
)
json_text = StringRequestCodec.decode_response(deserialized_response)
output = json.loads(json_text[0])
print(output)
A Python client converts a JSON message to UTF-8 bytes, wraps it in an inference request, converts it to the gRPC wire format, and opens a channel to the service. section 6.3.3
