Allreduce
Allreduce is a collective communication operation that combines corresponding values from participating processes or devices and distributes the result to every participant.
CUDA for LLMs
AllReduce is a collective communication operation that reduces values across participants and distributes the complete result to each participant.
- How it is used
- Combines gradients in data parallelism and partial matrix products in tensor parallelism, leaving each GPU with the result.
- How deep it goes
- Explains ring and tree phases and step counts, then applies an NCCL call and measures communication costs without implementing NCCL kernels.
- Examples
- Uses a four-GPU sum and an eight-GPU tensor-parallel matrix multiplication with measured timings.
- What you can do after
- Apply it via NCCL to combine partial GPU results, and evaluate its communication cost against end-to-end workload timing.
Distributed Machine Learning Patterns
An allreduce is a collective communication operation that aggregates values across participating processes and distributes the aggregate to every process.
- How it is used
- Synchronizes distributed training workers by aggregating gradients and keeping their model parameters identical.
- How deep it goes
- Explains reduce-and-broadcast composition, training rounds, scaling trade-offs, ring-allreduce, and recovery from stalled communication.
- Examples
- Uses gradient aggregation across worker-held model copies and a network slowdown that stalls a three-worker allreduce.
- What you can do after
- Evaluate/understand when allreduce can replace parameter servers and how to resume after slow or preempted workers.
Read how Distributed Machine Learning Patterns explains allreduce →
Deep Learning with JAX
A collective communication pattern that reduces values across devices and makes the result available to all participants.
- How it is used
- Coordinates shared reductions inside vectorized and device-parallel JAX computations.
- How deep it goes
- Explains named axes, reduction variants, grouped reductions, and multi-axis use through runnable normalization examples.
- Examples
- Demonstrates array normalization, grouped reductions, nested mapped axes, and sharded dot products requiring partial-result summation.
- What you can do after
- Apply it via JAX lax.psum, pmax, pmin, or pmean inside named pmap or vmap computations.
Deep Learning with PyTorch, Second Edition
A collective operation that combines values across participants and distributes the result to all of them.
- How it is used
- Coordinates values and gradients across processes during multi-GPU and distributed model training.
- How deep it goes
- Explains reduction semantics, rank-level results, PyTorch usage, and gradient placement, but does not detail communication algorithms or performance internals.
- Examples
- Demonstrates rank-held tensors, sum reduction to 3.0, and averaging synchronized gradients before parameter updates.
- What you can do after
- Apply it via torch.distributed.all_reduce to combine values or synchronize gradients across distributed training processes.
Read how Deep Learning with PyTorch, Second Edition explains allreduce →
Sutskever's List
An all-reduce operation combines values from multiple computing devices and distributes the combined result to all of them.
- How it is used
- Synchronizes replicated models in data-parallel training by combining gradients so each GPU applies a shared update.
- How deep it goes
- Defines gradient summation and synchronization, then notes GPU-transfer engineering and communication overhead without detailing the algorithm.
- Examples
- Uses DS2 training across 8–16 GPUs and compares its custom routine with standard OpenMPI.
- What you can do after
- Evaluate communication trade-offs in synchronized multi-GPU training, including diminishing returns as device counts rise.
Also covered in
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- collective communicationCollective communication is a coordinated operation in which multiple processes or devices exchange, combine, or synchronize data as part of a distributed computation.
- inner productAn inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
Related concepts
- collective communicationkind of · Collective communication is a coordinated operation in which multiple processes or devices exchange, combine, or synchronize data as part of a distributed computation.
- DistributedDataParallelused by · DistributedDataParallel is a PyTorch API that trains replicated models across processes or devices by synchronizing gradients during data-parallel computation.
- ring algorithmuses · A ring algorithm organizes participants in a circular communication sequence so each participant sends data to and receives data from a neighboring participant during collective communication.
- data parallelismused by · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
- all-gatherhas part · All-gather is a collective communication operation that gathers data from every participant and makes the combined result available to all participants.
- broadcast operationhas part · A broadcast operation is a collective communication operation that distributes data from one participant to all participants in a group.
Questions about allreduce
What is allreduce?
Allreduce is a collective communication operation that combines corresponding values from participating processes or devices and distributes the result to every participant.
Which Manning books explain allreduce?
CUDA for LLMs, Distributed Machine Learning Patterns, Deep Learning with JAX, Deep Learning with PyTorch, Second Edition and Sutskever's List.
What should I know before learning allreduce?
Manning books that teach allreduce usually introduce gradient, parameter, collective communication and inner product first.
What is allreduce a kind of?
Allreduce is a kind of collective communication. Collective communication is a coordinated operation in which multiple processes or devices exchange, combine, or synchronize data as part of a distributed computation.
What are the parts of allreduce?
Allreduce includes all-gather and broadcast operation.
What is allreduce also called?
Allreduce is also called all-reduce.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




