Parameter server pattern

As explained in Distributed Machine Learning Patterns

A parameter server is a server component that stores and updates a designated portion of a machine learning model's parameters.

Use the parameter server pattern to train a model whose parameters cannot fit on one machine. Dedicated parameter servers store and update designated model partitions, while worker nodes consume dataset partitions, calculate gradients, and send updates to the server holding the relevant partition. section 1.3.2 section 3.2.2

Figure 3.8 A machine learning training component with multiple parameter servers

Multiple parameter servers divide parameter storage, and workers asynchronously compute gradients from their assigned data before updating a model partition. section 3.2.2

A typical distributed flow is:

• Partition the model across one or more parameter servers when a single server cannot hold all parameters. section 3.2.2
• Assign each worker a subset of training data and let it perform neural-network calculations independently. section 3.2.2
• Send calculated gradients to the parameter server responsible for the target model partition. section 3.2.2
• Return updated model partitions to workers so later calculations can use current parameters. section 3.3.2

Large video-tagging model

A LeNet-based classifier for tagging new YouTube videos begins with one parameter server and multiple workers. As the model gains layers and parameters beyond one server’s capacity, split its parameters across additional servers; workers calculate gradients from data subsets and update the corresponding partitions. section 3.2.2 section 7.3.1

Asynchronous computation lets workers perform heavy calculations without continuously communicating with one another, but server communication can become the limiting cost. Concurrent gradient pushes and pulls can block; gradients computed from stale partitions may be dropped or retained depending on the cost of recomputation and their possible effect on the trained model. Uneven partition sizes can amplify the impact of stale updates. section 3.2.2 section 3.3.2 section 7.3.2

Choose this pattern for large, complex models that require distributed parameter storage. For small or medium-sized models that fit on a machine, communication with parameter servers may outweigh computation; collective communication instead lets every worker store a complete model copy and can improve training performance. section 3.4 section 3.6 section 7.3.2

How Distributed Machine Learning Patterns covers it

How deep it goes

Explains worker gradient flows, partitioned servers, asynchronous updates, blocking, stale gradients, and worker-to-server sizing trade-offs.

Examples

Uses LeNet video classification, multi-server model partitioning, and a 30 MB Fashion-MNIST model-selection case.

What it assumes

Assumes familiarity with machine-learning models, gradients, workers, datasets, and distributed training.

What you can do after

Evaluate/understand when partitioned parameter storage suits a large model and when communication overhead favors collective communication.

How it connects

Contrasts parameter servers with the collective communication pattern and relates them to distributed training and gradient updates.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Related concepts

• collective communicationalternative to · Collective communication is coordinated data exchange or synchronization among multiple processes or devices participating in a distributed computation.
• workerused by · A worker is a person or computational execution unit assigned to perform a task as part of a larger system or activity.
• TFJobconfigured by · TFJob is a Kubeflow custom resource that specifies the desired configuration and state of a TensorFlow training job, including distributed replicas.
• model partitionuses · A division of a machine-learning model or its parameters into separate portions assigned to distinct storage or computation components.
• distributed learning systemused by · A distributed learning system coordinates machine-learning computation, data processing, or model operations across multiple machines or execution nodes.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.