Distributed model serving system

As explained in Distributed Machine Learning Patterns

The process of serving a machine learning model across multiple networked machines, typically with large datasets or models partitioned or stored in distributed infrastructure.

A distributed model serving system generates predictions across a cluster when the trained model or serving dataset exceeds one machine’s capacity. It follows model training and must remain scalable, reliable, and efficient as request volume and request size grow. section 4.0 section 4.1

Core design concerns include:

• Store data in a remote distributed database or partition it across multiple machines. section 4.1
• Replicate services to handle increasing request volume through horizontal scaling. section 4.0
• Shard services to process large model-serving requests. section 4.0
• Assess whether event-driven design improves a real-world serving scenario. section 4.0

The design trades off scalability, reliability, efficiency, and communication overhead. Operational risks include unstable networks, corrupted data, and preempted machines, requiring deliberate distributed-system safeguards. section 4.0

Figure 4.1 A diagram showing where model serving fits in the machine learning pipeline

Locate model serving after model training within the machine-learning pipeline. section 4.1

Table 4.1 Comparison between traditional model serving and distributed model serving systems (view table figure)

	Traditional model serving	Distributed model serving
Computational resources	Personal laptop or single remote server	Cluster of machines
Dataset location	Local disk on a single laptop or machine	Remote distributed database or partitioned on disks of multiple machines
Size of model and dataset	Small enough to fit on a single machine	Medium to large

Compare single-machine and distributed serving by resources, storage location, and model or dataset size. section 4.1

How Distributed Machine Learning Patterns covers it

How deep it goes

Explains cluster placement, distributed storage, replication, sharding, event-driven assessment, and operational risks without implementation details.

Examples

Contrasts a laptop-hosted model and dataset with clustered serving backed by distributed storage or partitioned disks.

What it assumes

Assumes a successfully trained model and introduces serving as the subsequent prediction step, contrasting single-machine and distributed settings.

What you can do after

Evaluate serving architectures for scalability, reliability, efficiency, request size, and the suitability of event-driven design.

How it connects

Connects model training, data ingestion, distributed databases, replicated services, sharded services, event-driven design, and machine-learning workflows.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Related concepts

• Model Servingkind of · A model-serving platform is software infrastructure that hosts trained models and exposes inference by accepting requests and returning model outputs.
• scalabilityconfigures · Scalability is the ability of a system, process, or method to accommodate increasing workload, size, or demand while maintaining acceptable operation.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.