KServe is an open-source platform for standardized model serving on Kubernetes. It provides one inference interface across major machine-learning frameworks and serving backends, reducing the need to integrate each server separately. section 7.4.4 section 7.7
KServe organizes production serving around several infrastructure responsibilities:
- Accepts models from frameworks including TensorFlow, XGBoost, scikit-learn, PyTorch, and ONNX through an abstract serving interface. section 7.4.4
- Uses the KServe inference protocol so clients can query models hosted by systems such as Triton, TorchServe, and TensorFlow Serving through one API. section 7.4.4 section 7.7
- Uses Knative for Kubernetes network routing, worker autoscaling—including scale-to-zero—and model revision tracking. section 7.4.4
- Expects a model format, wrapper and configuration, repository location, and gRPC or HTTP/REST prediction endpoint. section 7.4.4
apiVersion: serving.kserve.io/v1beta1 #1
kind: InferenceService
metadata:
name: “torchserve”
spec:
predictor:
pytorch: #2
storageUri: gs://kfserving-examples/models #3
/torchserve/image_classifier #3
A compact InferenceService configuration selects a PyTorch predictor and points it to a model in object storage. section 7.4.4
