Backend predictor

As explained in Designing Deep Learning Systems: A guide for software engineers

A model-specific software component that receives serving inputs and executes prediction logic for a model service.

A backend predictor is the model-specific execution component behind a prediction API. It receives forwarded inference requests, loads model files, executes prediction logic, and returns results. section 7.1.2 section 7.5.2

In a model service, the frontend owns the public API and model-file retrieval, while the predictor reads models from a shared disk volume. A self-built predictor can support several models of one model family. section 7.1 section 7.1.2

A typical request workflow is:

• Accept a request containing a model identifier and input text. section 7.1.2
• Load the selected model from shared storage into memory. section 7.1.2
• Preprocess inputs, execute the model, postprocess outputs, and return the prediction. section 7.5.2

Runtime model selection

A prediction request can specify a model name and version or a model ID. The routing layer consults model metadata, downloads the files, selects the appropriate predictor by algorithm type, and forwards the request. section 7.5.2

Figure 7.1 A system overview and model serving end-to-end workflow

Trace the frontend, metadata store, shared volume, and backend predictor through an end-to-end serving request. section 7.1.2

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Explains responsibilities and request flow through loading, preprocessing, execution, and postprocessing, without implementation code.

Examples

Uses intent classification models, model IDs, model versions, cloud-stored files, and shared-volume loading workflows.

What it assumes

Assumes model-serving components and metadata exist, while explaining how the frontend, storage, routing, and predictor cooperate.

What you can do after

Build a model-specific serving container that loads models and executes predictions behind a frontend API.

How it connects

Connects frontend APIs, metadata stores, shared disk volumes, model services, model servers, and runtime model loading.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Related concepts

• intent classificationuses · Intent classification is the task of assigning an input, such as a query or utterance, to a category representing its purpose, goal, or information need.
• Model Servingpart of · A model-serving platform is software infrastructure that hosts trained models and exposes inference by accepting requests and returning model outputs.
• model service approachused by · A model-serving pattern in which application-specific code contains the model prediction workflow and directly serves a particular model type.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.