Prediction platform

As explained in Designing Deep Learning Systems: A guide for software engineers

A platform that exposes shared infrastructure for serving multiple applications and different types of machine learning models.

A prediction platform is a centralized model-serving system that exposes shared infrastructure for multiple applications, model types, algorithms, frameworks, and model versions. Consolidating onboarding and maintenance in one system can cost less than operating a separate prediction service for each application. section 6.3.3 section 6.4

Figure 6.7 A general prediction service (platform) design that works with arbitrary model types

Use the reference architecture to trace how a unified API, routing, graph execution, and model-server backends organize serving for arbitrary model types. section 6.3.3

A platform brings several serving responsibilities into one system:

• Expose a generic prediction API whose request and response schemas can support different model algorithms; KFServing’s predict protocol is one example. section 6.3.3
• Coordinate routing, graph execution, and multiple model-server backends. section 6.4
• Account for model file formats, libraries, training frameworks, caching, versioning, data processing, and model management. section 6.3.3

The approach fits systems serving multiple applications with different models, such as chatbot, face-swapping, flower-recognition, PDF-scanning, and voice-recognition workloads. Choose it alongside the model server approach when serving requirements span applications and model technologies. section 6.3.3 section 6.4

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Defines the architecture and enumerates lifecycle components without detailing their individual implementations.

Examples

Uses chatbot, face-swapping, flower recognition, PDF scanning, and voice recognition as application scenarios.

What it assumes

Starts from the serving use case and distinguishes single-model, multitenant, and multi-application requirements.

What you can do after

Evaluate serving approaches by application count, model diversity, and deployment, maintenance, monitoring, infrastructure, and development costs.

How it connects

Connects model serving with model servers, unified prediction APIs, routing, graph execution, model caching, and model versioning.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Related concepts

• Model Servingkind of · A model-serving platform is software infrastructure that hosts trained models and exposes inference by accepting requests and returning model outputs.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.