The model server approach centralizes prediction behind a general-purpose backend. Rather than building a dedicated predictor for each model type, an application sends requests to a model server that loads and executes supported models through a common serving system. This separates model execution from application-specific frontend APIs and reduces the dependency between training code and serving code. section 7.3 section 7.1.4
It is best suited to a model serving platform that must host many model types. One predictor backend can support the platform’s models, avoiding separate backend implementations and their associated development and maintenance effort. The trade-off is operational complexity: teams must learn, configure, operate, and maintain the server system rather than owning a small end-to-end predictor implementation. section 7.3 section 7.7
A typical deployment divides responsibility among these parts:
- A frontend API receives prediction requests and delegates model execution to the backend. section 7.7
- The model server hosts model artifacts and executes predictions for compatible model formats and frameworks. section 7.7
- A serving interface can standardize access across server backends, allowing one API to work with TensorFlow Serving, TorchServe, and Triton. section 7.7
- Model release design relates model versions to the serving system and should account for production operation and monitoring. section 7.7
TorchServe-backed intent API
An intent-classification service can retain its frontend API while replacing a custom predictor container with a TorchServe predictor. The backend is then not confined to intent classification and can serve arbitrary PyTorch models, illustrating why a reusable server backend helps when model types multiply. section 7.0 section 7.7
Use an existing server when possible instead of building one. TorchServe, TensorFlow Serving, and Triton follow this approach; Triton is recommended where applicable for broad training-framework compatibility and GPU-acceleration performance. For a single application with one straightforward model type, a custom model service may remain simpler to build and maintain. section 7.7 section 7.3
