Allreduce

As explained in Distributed Machine Learning Patterns

An allreduce is a collective communication operation that aggregates values across participating processes and distributes the aggregate to every process.

Allreduce is a collective communication operation that combines values from every process with a chosen reduce function, then distributes the aggregate back to every process. In distributed model training, workers use it to aggregate gradients so every local model receives the same update. section 3.3.2

Figure 3.20 An example of an allreduce operation that reduces the results on each process in the group and then sends the result to every process in the group

The operation first reduces values held across the process group, then makes the resulting value available to every worker. section 3.3.2

A synchronous training round follows this flow:

• Each worker consumes a data partition and calculates gradients against its local model copy. section 3.3.2 section 7.3.2
• Allreduce aggregates gradients across workers and returns the aggregate to each worker. section 3.3.2
• Every worker updates its parameters from the same aggregate, preserving identical model copies before the next round. section 3.3.2

This worker-only pattern can replace parameter servers for models that are not too large, removing worker-to-parameter-server communication and server-to-worker ratio tuning. It remains network-bound: basic allreduce requires each worker to communicate with peers, and a large worker count can slow training. Ring-allreduce reduces peer connections to two per worker and is bandwidth-optimal when aggregated gradients are sufficiently large. section 3.3.3

Recovering from a stalled round

Allreduce is synchronous: a slow network communication can prevent the whole process group from completing. Abandon the affected workers and current operation, reconstruct a group from the remaining workers, and resume allreduce because those workers retain the same model copy. The same approach handles preempted workers. section 3.4.2

In a multi-model training architecture, each training step uses allreduce over the underlying network to communicate gradients among workers. After training, a separate process can select a top model for serving. section 7.3.2

How Distributed Machine Learning Patterns covers it

How deep it goes

Explains reduce-and-broadcast composition, training rounds, scaling trade-offs, ring-allreduce, and recovery from stalled communication.

Examples

Uses gradient aggregation across worker-held model copies and a network slowdown that stalls a three-worker allreduce.

What it assumes

Builds from reduce and broadcast operations while framing their use through worker gradients and local model copies.

What you can do after

Evaluate/understand when allreduce can replace parameter servers and how to resume after slow or preempted workers.

How it connects

Connects collective communication with reduce, broadcast, parameter servers, ring-allreduce, synchronous training, and worker process groups.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with JAXA collective communication pattern that reduces values across devices and makes the result available to all participants.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA collective operation that combines values across participants and distributes the result to all of them.
• 

  liveBookmanning.com
  Sutskever's ListAn all-reduce operation combines values from multiple computing devices and distributes the combined result to all of them.

Compare how 4 books explain allreduce →

Related concepts

• DistributedDataParallelused by · DistributedDataParallel is a PyTorch module that replicates a model across processes or devices and synchronizes gradients during data-parallel training.
• data parallelused by · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
• collective communicationused by · Collective communication is coordinated data exchange or synchronization among multiple processes or devices participating in a distributed computation.
• collective operationkind of · A collective operation is a coordinated parallel communication primitive in which participating processes or devices exchange or combine data.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.