Model service approach

As explained in Designing Deep Learning Systems: A guide for software engineers

A model-serving pattern in which application-specific code contains the model prediction workflow and directly serves a particular model type.

The model service approach packages one model’s prediction workflow as a dedicated web service. An application backend sends requests to a predictor, which loads a model artifact, preprocesses input, runs inference, postprocesses output, and returns the result. The predictor commonly runs in a Docker container and can often be adapted from the training container. section 6.3.1

Figure 6.4 A single model predictor design in a client/server setup

A client/server layout separates the mobile UI, application backend, and single-model predictor. The backend relays a user request to the predictor rather than executing the model in the UI. section 6.3.1

Face-swap predictor

A phone UI uploads source and target images and asks an application backend to swap faces. The backend calls a deepfake predictor API; the predictor transforms the images, executes the model, and returns swapped images for display. This layout suits an application with a web backend and one model. section 6.3.1

This pattern is a pragmatic fit for a single application. Dedicated predictor code is relatively quick to build because serving code simplifies training code, and owning the end-to-end workflow can make maintenance straightforward. Its specificity is also its limit: a predictor built from intent-classification training code serves that model type rather than arbitrary models. section 7.3 section 7.7

Choose the pattern when its operational trade-offs match the system:

• Use it for a single application or a small, model-specific workflow where control of preprocessing, inference, and postprocessing matters. section 7.3 section 7.7
• Prefer a model server for a platform that must support many model types; one generic predictor backend avoids building a dedicated service for every model. section 7.3 section 7.7
• Plan releases and monitoring alongside the service design, including model metric collection and model quality gates. section 7.7

A self-built serving container demonstrates the approach, while TorchServe illustrates the contrasting model server approach. Model servers can serve arbitrary models within their supported ecosystem, but their setup, operation, and maintenance require learning a broader serving system. Existing server tools are recommended over building a model server from scratch when that broader platform capability is needed. section 7.0 section 7.3 section 7.7

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Traces request handling through preprocessing, inference, postprocessing, model loading, and container deployment without detailing model internals.

Examples

Demonstrates face swapping through a mobile UI, application backend, and deepfake predictor; also uses an intent-classification predictor.

What it assumes

Assumes familiarity with model training code and web-backend applications while presenting the serving workflow concretely.

What you can do after

Evaluate/understand when to build a dedicated predictor versus adopt a model server for a serving platform.

How it connects

Contrasts dedicated predictors with model servers, Docker containers, TorchServe, model artifacts, model monitoring, and quality gates.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Related concepts

• direct model embeddingalternative to · A model-serving design in which model execution code is integrated with application logic and runs in the same process.
• containeruses · A software container is an isolated execution environment that packages an application and its dependencies while sharing the host operating system kernel.
• intent classificationuses · Intent classification is the task of assigning an input, such as a query or utterance, to a category representing its purpose, goal, or information need.
• web serviceuses · A web service is a programmatic software interface that exposes functionality or data through web technologies or network protocols, typically HTTP.
• Model Servingimplements · A model-serving platform is software infrastructure that hosts trained models and exposes inference by accepting requests and returning model outputs.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.