Predictor backend client

As explained in Designing Deep Learning Systems: A guide for software engineers

A client component that represents and communicates with a backend predictor service, including transferring models or prediction requests.

A predictor backend client represents one backend predictor and connects the frontend service to it. It manages backend communication and can transfer model files before forwarding prediction requests. section 7.1.3 section 7.2.3

A backend client participates in the serving workflow by:

• Implementing model operations such as downloadModel, registerModel, and predict. section 7.1.3
• Receiving the selected model metadata and checking or downloading its files to shared storage. section 7.1.3
• Sending the model run ID and document to the registered predictor through a gRPC stub. section 7.1.3
• Supporting multiple backend technologies by registering clients under model algorithm types. section 7.1.3 section 7.2.3

Intent classification with a custom gRPC backend

A request for model A reaches the web interface, which obtains metadata and asks the predictor connection manager to select the client for the model algorithm. CustomGrpcPredictorBackend downloads the model if necessary, invokes the registered intent predictor, and returns its response. section 7.1.3

Adding a TorchServe client

Register TorchGrpcPredictorBackend for models whose metadata names TorchServe. The client downloads model files and sends prediction requests to the TorchServe container. section 7.2.3

Figure 7.2 The frontend service design and the model serving workflow

Use the workflow diagram to trace request routing, metadata lookup, backend selection, model transfer, and prediction communication. section 7.1.3

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Explains responsibilities, configuration, request flow, and interface methods, but defers predictor implementation details.

Examples

Uses CustomGrpcPredictorBackend for intent classification and TorchGrpcPredictorBackend for TorchServe.

What it assumes

Assumes frontend-service, model-metadata, shared-storage, and gRPC concepts while defining the backend interface from its responsibilities.

What you can do after

Operate/configure backend routing by registering algorithm mappings, model locations, predictor hosts, ports, and technology types.

How it connects

Connects frontend services, predictor connection managers, metadata stores, model serving, gRPC, and TorchServe.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.