Model partition

As explained in Distributed Machine Learning Patterns

A subset of a machine learning model's parameters assigned to a particular storage or computation component.

A model partition is a subset of model parameters stored and updated by a particular parameter server. Partitioning lets distributed training handle models whose full parameter set cannot fit on one machine. section 1.3.2 section 3.2.2

A multi-server training workflow assigns responsibilities across workers and parameter servers:

• Workers consume separate dataset portions, calculate neural-network-layer results and gradients, and send updates to the relevant parameter server. section 3.2.2 section 7.3.2
• Parameter servers store and update designated model partitions, avoiding the need for one server to hold the complete model. section 1.3.2 section 3.2.2
• Asynchronous workers may use stale partitions, so systems continually push and pull parameter updates. section 3.2.2
• Balance worker and parameter-server counts carefully: simultaneous gradient traffic can cause blocking communication and create a bottleneck. section 7.3.2

Large-scale video classification

A model tagging unseen YouTube videos can distribute its parameters across multiple parameter servers. Workers process video-data subsets and update the partitions holding the model’s parameters, enabling training beyond a single machine’s capacity. section 3.2.2

Figure 3.8 A machine learning training component with multiple parameter servers

Use the architecture diagram to trace workers, multiple parameter servers, model partitions, and gradient updates. section 3.2.2

How Distributed Machine Learning Patterns covers it

How deep it goes

Explains storage, gradient updates, stale parameters, synchronization, blocking, and worker-server sizing, but does not specify implementation code.

Examples

Uses YouTube video classification and multi-parameter-server architecture diagrams to illustrate partitioned training.

What it assumes

Assumes familiarity with workers, datasets, model parameters, gradients, and distributed training; introduces partitioned parameter storage through concrete flows.

What you can do after

Evaluate when partitioned parameter-server training can fit and efficiently train models that exceed a single machine’s capacity.

How it connects

Connects parameter servers, distributed model training, data partitions, asynchronous computation, gradient updates, and collective communication.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Related concepts

• parameter server patternused by · The parameter server pattern is a distributed-training pattern in which dedicated servers store and update partitions of a model's parameters.
• checkpointingused by · A checkpoint is a designated point in a process at which its state or progress is recorded or evaluated, often to support recovery, continuation, or control.
• repartitionused by · Repartitioning is the redistribution of records or items across partitions according to a partitioning rule, potentially changing the number of partitions.
• parameterpart of · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
• partitionkind of · A partition is a distinct subdivision of a larger data, storage, or work unit that can be stored, managed, or processed separately.
• language modelpart of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.