Web predict API

As explained in Designing Deep Learning Systems: A guide for software engineers

An interface for submitting requests that execute a model and return its prediction result.

A web predict API is the request boundary between an application and a model-serving predictor. The predictor loads a model file from a model artifact repository, executes the model, and returns the prediction result. In a single-model application, the predictor commonly runs as a web service in a Docker container. section 6.3.1

A production-oriented design assigns the API these responsibilities:

• Accept prediction requests from the web backend and pass them to the predictor. section 6.3.1
• Use a generic request and response payload schema when the service must support arbitrary models. section 6.3.3
• Work alongside metadata and deployment APIs that expose model version, algorithm information, and deployment status. section 6.3.3
• Allow a routing component to select an inference backend by matching model metadata with routing configuration. section 6.3.3

Single-model predictor

Convert a model-training container into a predictor container by adding the web predict API and the neural network's evaluation mode. The resulting service loads the trained model and responds to application requests. section 6.3.1

Figure 6.4 A single model predictor design in a client/server setup

Use the single-model predictor architecture to trace the client/server request path and the predictor's role. section 6.3.1

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Explains service placement, generic payloads, related APIs, and backend routing, but does not specify concrete endpoint syntax.

Examples

Uses a Dockerized single-model predictor and a generic prediction protocol such as KFServing's predict protocol.

What it assumes

Assumes a trained model, a model artifact repository, and a web backend while introducing the predictor boundary.

What you can do after

Build it from scratch by adapting a training container into a predictor service with an API and evaluation mode.

How it connects

Connects model serving, model metadata APIs, model deployment APIs, routing components, and inference backends.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.