Collective communication
Collective communication is a coordinated operation in which multiple processes or devices exchange, combine, or synchronize data as part of a distributed computation.
Distributed Machine Learning Patterns
A collective communication pattern coordinates communication of computation updates among multiple workers and synchronizes their model copies.
- How it is used
- Speeds distributed training of small or medium-sized models when parameter-server communication becomes a bottleneck.
- How deep it goes
- Explains worker-only architecture, reduce, broadcast, allreduce, and ring-allreduce, while deferring implementation details.
- Examples
- Uses contending workers and parameter servers, plus a 30 MB Fashion-MNIST classification model with worker-only allreduce.
- What you can do after
- Evaluate/understand when worker-only allreduce can replace parameter servers for distributed model training.
Read how Distributed Machine Learning Patterns explains collective communication →
CUDA for LLMs
A collective operation is a coordinated communication operation involving a group of distributed processes or devices.
- How it is used
- Coordinates GPU data movement and combines partial results in distributed training and tensor-parallel computation.
- How deep it goes
- Defines major communication patterns and explains topology and scaling tradeoffs, while leaving collective kernel implementations to NCCL.
- Examples
- Uses four-GPU gradient summation and an eight-GPU matrix multiply with measured AllReduce overhead.
- What you can do after
- Evaluate collective communication costs using topology, message size, bandwidth benchmarks, and end-to-end workload measurements.
Deep Learning with PyTorch, Second Edition
A distributed communication pattern in which multiple workers exchange or synchronize data.
- How it is used
- Synchronizes data and gradients across devices during distributed training.
- How deep it goes
- Defines broadcast and allreduce with examples, while treating the wider collective set only as a brief extension.
- Examples
- Demonstrates rank-zero broadcast of 123.0 and rank-based allreduce summation using torch.distributed.
- What you can do after
- Apply it via torch.distributed to broadcast tensors and reduce shared values across processes.
Read how Deep Learning with PyTorch, Second Edition explains collective communication →
Also covered in
- Deep Learning with JAX
- GPU Programming with TritonIn early access (MEAP)
What to know first
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- parallelismParallelism is the execution of multiple independent tasks or computational operations at the same time rather than sequentially.
- data parallelismData parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
Related concepts
- allreduceincludes · Allreduce is a collective communication operation that combines corresponding values from participating processes or devices and distributes the result to every participant.
- NCCLimplemented by · NCCL is NVIDIA's library of optimized collective communication primitives for multi-GPU and multi-node computing.
- axis nameused by · Labels assigned to mapped axes in JAX so collective operations can refer to the device or batch dimension they act over.
- parameter server patternalternative to · The parameter server pattern is a distributed-training pattern in which dedicated servers store and update partitions of a model's parameters.
- reduction operationincludes · A reduction operation combines multiple input values according to a rule to produce fewer results, often a single aggregate.
- trainingused by · Training is the process of fitting a machine-learning model to data by adjusting its parameters or learned representations to improve its performance on a task.
Questions about collective communication
What is collective communication?
Collective communication is a coordinated operation in which multiple processes or devices exchange, combine, or synchronize data as part of a distributed computation.
Which Manning books explain collective communication?
Distributed Machine Learning Patterns, CUDA for LLMs and Deep Learning with PyTorch, Second Edition.
What should I know before learning collective communication?
Manning books that teach collective communication usually introduce gradient, parameter, parallelism and data parallelism first.
What is the difference between collective communication and parameter server pattern?
Collective communication is a coordinated operation in which multiple processes or devices exchange, combine, or synchronize data as part of a distributed computation. The parameter server pattern is a distributed-training pattern in which dedicated servers store and update partitions of a model's parameters.
What kinds of collective communication are there?
Kinds of collective communication include allreduce and reduction operation.
What is collective communication also called?
Collective communication is also called collective operation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


