Allreduce
Allreduce is a collective communication operation that combines corresponding values across participating processes or devices and distributes the resulting aggregate to every participant.
Distributed Machine Learning Patterns
An allreduce is a collective communication operation that aggregates values across participating processes and distributes the aggregate to every process.
- How it is used
- Synchronizes distributed training workers by aggregating gradients and keeping their model parameters identical.
- How deep it goes
- Explains reduce-and-broadcast composition, training rounds, scaling trade-offs, ring-allreduce, and recovery from stalled communication.
- Examples
- Uses gradient aggregation across worker-held model copies and a network slowdown that stalls a three-worker allreduce.
- What you can do after
- Evaluate/understand when allreduce can replace parameter servers and how to resume after slow or preempted workers.
Read how Distributed Machine Learning Patterns explains allreduce →
Deep Learning with JAX
A collective communication pattern that reduces values across devices and makes the result available to all participants.
- How it is used
- Coordinates shared reductions inside vectorized and device-parallel JAX computations.
- How deep it goes
- Explains named axes, reduction variants, grouped reductions, and multi-axis use through runnable normalization examples.
- Examples
- Demonstrates array normalization, grouped reductions, nested mapped axes, and sharded dot products requiring partial-result summation.
- What you can do after
- Apply it via JAX lax.psum, pmax, pmin, or pmean inside named pmap or vmap computations.
Deep Learning with PyTorch, Second Edition
A collective operation that combines values across participants and distributes the result to all of them.
- How it is used
- Coordinates values and gradients across processes during multi-GPU and distributed model training.
- How deep it goes
- Explains reduction semantics, rank-level results, PyTorch usage, and gradient placement, but does not detail communication algorithms or performance internals.
- Examples
- Demonstrates rank-held tensors, sum reduction to 3.0, and averaging synchronized gradients before parameter updates.
- What you can do after
- Apply it via torch.distributed.all_reduce to combine values or synchronize gradients across distributed training processes.
Read how Deep Learning with PyTorch, Second Edition explains allreduce →
Sutskever's List
An all-reduce operation combines values from multiple computing devices and distributes the combined result to all of them.
- How it is used
- Synchronizes replicated models in data-parallel training by combining gradients so each GPU applies a shared update.
- How deep it goes
- Defines gradient summation and synchronization, then notes GPU-transfer engineering and communication overhead without detailing the algorithm.
- Examples
- Uses DS2 training across 8–16 GPUs and compares its custom routine with standard OpenMPI.
- What you can do after
- Evaluate communication trade-offs in synchronized multi-GPU training, including diminishing returns as device counts rise.
Related concepts
- DistributedDataParallelused by · DistributedDataParallel is a PyTorch module that replicates a model across processes or devices and synchronizes gradients during data-parallel training.
- data parallelused by · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
- collective communicationused by · Collective communication is coordinated data exchange or synchronization among multiple processes or devices participating in a distributed computation.
- collective operationkind of · A collective operation is a coordinated parallel communication primitive in which participating processes or devices exchange or combine data.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.