Distributed learning system

As explained in Distributed Machine Learning Patterns

A distributed machine learning system is a machine learning system whose computation, data processing, or model operations are distributed across multiple machines or execution nodes.

A distributed learning system is a machine learning pipeline whose components run across multiple machines or execution nodes. Its components can handle data ingestion, training, serving, versioning, and monitoring, allowing large datasets, large models, heavy serving traffic, and complex model selection workloads to be handled more scalably and reliably. section 1.3.1 section 1.3.3 section 1.5

Distribution is warranted when one machine cannot store the model or serve the traffic, when training needs more workers or hardware acceleration, or when the model lifecycle requires coordinated components. It also adds design, maintenance, and operational overhead; a small dataset, simple model, or adequate single-machine resources favor a simpler approach. section 1.3.3 section 1.3.4

Common ways to distribute work include:

• Partition data so multiple workers consume separate portions during training. section 1.3.2
• Use asynchronous workers in a work-queue-style design to speed training through parallel data consumption. section 1.3.2
• Use parameter servers to store and update model partitions when a model cannot fit on one server. section 1.3.2
• Replicate model servers for horizontal scaling, shard services for large requests, and evaluate event-driven serving where appropriate. section 7.0

Figure 1.5 An example of applying the work-queue pattern in distributed machine learning systems

The work-queue view makes data-parallel training concrete: workers take data partitions from shared storage and use them to update a centralized model. section 1.3.2

End-to-end pipeline

A practical distributed pipeline connects ingestion and preprocessing to distributed training, model serving, workflows, and operational automation. TensorFlow with Python builds models; Kubernetes manages cluster resources; Kubeflow submits distributed jobs; Argo Workflows orchestrates components; and Docker supplies container images. section 1.4 section 7.0

Training is the central distributed component: designs must handle models and datasets of different sizes while addressing communication overhead, performance bottlenecks, corrupted data, and cluster or network failures. Parameter servers suit models too large for one machine, while collective communication can improve training performance for smaller models. section 3.0 section 1.3.2

How Distributed Machine Learning Patterns covers it

How deep it goes

Defines pipeline components and distribution patterns, then defers detailed implementation to training, serving, workflow, and operations patterns.

Examples

Illustrates data-partition workers, parameter servers, and an end-to-end TensorFlow, Kubernetes, Kubeflow, Argo Workflows, and Docker pipeline.

What it assumes

Assumes Python ML-model experience, production exposure, and roughly one year of Python and Bash; does not require distributed-systems experience.

What you can do after

Apply distributed patterns and common tooling to build, deploy, manage, and automate ML workflows.

How it connects

Connects data ingestion, distributed training, model serving, machine learning workflows, parameter servers, and collective communication.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Related concepts

• distributed systemkind of · A distributed system is a computing system whose components run across multiple networked processes or machines and coordinate through communication.
• parameter server patternuses · The parameter server pattern is a distributed-training pattern in which dedicated servers store and update partitions of a model's parameters.
• machine learning systemkind of · A machine learning system is a software system that uses models trained on data, together with associated data, computation, and operational components, to perform tasks.
• pipelinehas part · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
• data ingestionhas part · Data ingestion is the process of acquiring or receiving data from one or more sources and bringing it into a system for storage or processing.
• traininghas part · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.