A predictor connection manager coordinates model-serving backends inside a frontend service. It registers predictor clients, looks up model metadata, and selects a backend by algorithm type. section 7.1.3 section 7.2.3
A practical implementation gives the manager several responsibilities:
- Register each algorithm and its backend properties, including host, port, and technology stack. section 7.1.3
- Select the backend matching the model metadata's algorithm type before downloading model files or invoking prediction. section 7.1.3
Routing an intent-classification request
A request carrying a model run ID reaches the prediction API. The manager obtains artifact metadata, finds the intent-classification mapping, and selects CustomGrpcPredictorBackend. That client downloads the model from its URL when necessary, registers it with the configured predictor, and sends the document and run ID through gRPC. Adding TorchServe follows the same pattern: register TorchGrpcPredictorBackend, then route models whose metadata identifies TorchServe. section 7.1.3 section 7.2.3
