Routing component

As explained in Designing Deep Learning Systems: A guide for software engineers

A serving-system component that selects an inference backend for a prediction request according to model metadata and routing configuration.

A routing component selects the inference backend for each prediction request. Because serving backends usually support only particular model types, routing enables one prediction platform to serve models across different frameworks through a unified API. section 6.3.3 section 6.4

For each prediction request, the component:

• Fetches the model metadata, including the algorithm name, model version, training framework, and metadata version information. section 6.3.3
• Matches that metadata against routing configuration to identify a compatible inference backend. section 6.3.3
• Forwards the prediction request to the selected inference server. section 6.3.3

Selecting framework-specific servers

A platform can route TensorFlow models to TensorFlow Serving and PyTorch models to TorchServe. The client uses the common prediction interface while the routing component chooses the framework-specific backend. section 6.3.3

Figure 6.7 A general prediction service (platform) design that works with arbitrary model types

Use the general prediction-service architecture to locate the routing component between the unified prediction API and multiple model-server backends. section 6.3.3

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Defines metadata lookup, configuration matching, and request forwarding, but does not detail routing algorithms or backend implementation internals.

Examples

Uses TensorFlow Serving for TensorFlow models and TorchServe for PyTorch models as framework-specific inference backends.

What it assumes

Assumes model-serving concepts, prediction requests, model metadata, and multiple inference backends.

What you can do after

Operate/configure prediction routing by matching model metadata with backend-selection rules.

How it connects

Connects unified prediction APIs, model metadata, routing configuration, inference backends, and model servers.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Related concepts

• Model Servinguses · A model-serving platform is software infrastructure that hosts trained models and exposes inference by accepting requests and returning model outputs.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.