liveBook · Manning

Allreduce

Also called all-reduce

Allreduce is a collective communication operation that combines corresponding values from participating processes or devices and distributes the result to every participant.

How 5 Manning books explain it

CUDA for LLMs

AllReduce is a collective communication operation that reduces values across participants and distributes the complete result to each participant.

How it is used
Combines gradients in data parallelism and partial matrix products in tensor parallelism, leaving each GPU with the result.
How deep it goes
Explains ring and tree phases and step counts, then applies an NCCL call and measures communication costs without implementing NCCL kernels.
Examples
Uses a four-GPU sum and an eight-GPU tensor-parallel matrix multiplication with measured timings.
What you can do after
Apply it via NCCL to combine partial GPU results, and evaluate its communication cost against end-to-end workload timing.

Read how CUDA for LLMs explains allreduce →

Distributed Machine Learning Patterns

An allreduce is a collective communication operation that aggregates values across participating processes and distributes the aggregate to every process.

How it is used
Synchronizes distributed training workers by aggregating gradients and keeping their model parameters identical.
How deep it goes
Explains reduce-and-broadcast composition, training rounds, scaling trade-offs, ring-allreduce, and recovery from stalled communication.
Examples
Uses gradient aggregation across worker-held model copies and a network slowdown that stalls a three-worker allreduce.
What you can do after
Evaluate/understand when allreduce can replace parameter servers and how to resume after slow or preempted workers.

Read how Distributed Machine Learning Patterns explains allreduce →

Deep Learning with JAX

A collective communication pattern that reduces values across devices and makes the result available to all participants.

How it is used
Coordinates shared reductions inside vectorized and device-parallel JAX computations.
How deep it goes
Explains named axes, reduction variants, grouped reductions, and multi-axis use through runnable normalization examples.
Examples
Demonstrates array normalization, grouped reductions, nested mapped axes, and sharded dot products requiring partial-result summation.
What you can do after
Apply it via JAX lax.psum, pmax, pmin, or pmean inside named pmap or vmap computations.

Read how Deep Learning with JAX explains allreduce →

Deep Learning with PyTorch, Second Edition

A collective operation that combines values across participants and distributes the result to all of them.

How it is used
Coordinates values and gradients across processes during multi-GPU and distributed model training.
How deep it goes
Explains reduction semantics, rank-level results, PyTorch usage, and gradient placement, but does not detail communication algorithms or performance internals.
Examples
Demonstrates rank-held tensors, sum reduction to 3.0, and averaging synchronized gradients before parameter updates.
What you can do after
Apply it via torch.distributed.all_reduce to combine values or synchronize gradients across distributed training processes.

Read how Deep Learning with PyTorch, Second Edition explains allreduce →

Sutskever's List

An all-reduce operation combines values from multiple computing devices and distributes the combined result to all of them.

How it is used
Synchronizes replicated models in data-parallel training by combining gradients so each GPU applies a shared update.
How deep it goes
Defines gradient summation and synchronization, then notes GPU-transfer engineering and communication overhead without detailing the algorithm.
Examples
Uses DS2 training across 8–16 GPUs and compares its custom routine with standard OpenMPI.
What you can do after
Evaluate communication trade-offs in synchronized multi-GPU training, including diminishing returns as device counts rise.

Read how Sutskever's List explains allreduce →

Also covered in

What to know first

Related concepts

Questions about allreduce

What is allreduce?

Allreduce is a collective communication operation that combines corresponding values from participating processes or devices and distributes the result to every participant.

Which Manning books explain allreduce?

CUDA for LLMs, Distributed Machine Learning Patterns, Deep Learning with JAX, Deep Learning with PyTorch, Second Edition and Sutskever's List.

What should I know before learning allreduce?

Manning books that teach allreduce usually introduce gradient, parameter, collective communication and inner product first.

What is allreduce a kind of?

Allreduce is a kind of collective communication. Collective communication is a coordinated operation in which multiple processes or devices exchange, combine, or synchronize data as part of a distributed computation.

What are the parts of allreduce?

Allreduce includes all-gather and broadcast operation.

What is allreduce also called?

Allreduce is also called all-reduce.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.