Allreduce

As explained in Deep Learning with PyTorch, Second Edition

A collective operation that combines values across participants and distributes the result to all of them.

Allreduce is a collective operation in which every process contributes data, the processes combine it with a reduction such as sum or max, and every process receives the result. For example, ranks holding t0, t1, t2, and t3 all receive their combined sum. section 16.2

In distributed training, use allreduce after backpropagation and before the optimizer step to aggregate or average gradients across devices. Synchronized gradients let each replica update consistently instead of drifting into separate models. section 16.3

Summing values across ranks

With PyTorch, each rank can create tensor = torch.tensor([float(rank)]) and call dist.all_reduce(tensor, op=dist.ReduceOp.SUM). Every process then holds the sum; with ranks contributing 0, 1, and 2, the shared result is 3.0. Process output order may vary because multiprocessing is nondeterministic. section 16.2

Figure 16.3 Illustration of the broadcast (left) and allreduce (right) collective communication operations

Visualizes how allreduce combines each rank’s tensor and distributes the result back to every rank. section 16.2

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes

Explains reduction semantics, rank-level results, PyTorch usage, and gradient placement, but does not detail communication algorithms or performance internals.

Examples

Demonstrates rank-held tensors, sum reduction to 3.0, and averaging synchronized gradients before parameter updates.

What it assumes

Assumes processes, ranks, tensors, backward passes, gradients, and optimizer steps; introduces collective communication through accessible comparisons.

What you can do after

Apply it via torch.distributed.all_reduce to combine values or synchronize gradients across distributed training processes.

How it connects

Connects collective communication with broadcast, gradient reduction, distributed training, data parallelism, and optimizer steps.

liveBookmanning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Distributed Machine Learning PatternsAn allreduce is a collective communication operation that aggregates values across participating processes and distributes the aggregate to every process.
• 

  liveBookmanning.com
  Deep Learning with JAXA collective communication pattern that reduces values across devices and makes the result available to all participants.
• 

  liveBookmanning.com
  Sutskever's ListAn all-reduce operation combines values from multiple computing devices and distributes the combined result to all of them.

Compare how 4 books explain allreduce →

Related concepts

• DistributedDataParallelused by · DistributedDataParallel is a PyTorch module that replicates a model across processes or devices and synchronizes gradients during data-parallel training.
• data parallelused by · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
• collective communicationused by · Collective communication is coordinated data exchange or synchronization among multiple processes or devices participating in a distributed computation.
• collective operationkind of · A collective operation is a coordinated parallel communication primitive in which participating processes or devices exchange or combine data.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.