Collective communication
Collective communication is coordinated data exchange or synchronization among multiple processes or devices participating in a distributed computation.
Distributed Machine Learning Patterns
A collective communication pattern coordinates communication of computation updates among multiple workers and synchronizes their model copies.
- How it is used
- Speeds distributed training of small or medium-sized models when parameter-server communication becomes a bottleneck.
- How deep it goes
- Explains worker-only architecture, reduce, broadcast, allreduce, and ring-allreduce, while deferring implementation details.
- Examples
- Uses contending workers and parameter servers, plus a 30 MB Fashion-MNIST classification model with worker-only allreduce.
- What you can do after
- Evaluate/understand when worker-only allreduce can replace parameter servers for distributed model training.
Read how Distributed Machine Learning Patterns explains collective communication →
Deep Learning with PyTorch, Second Edition
A distributed communication pattern in which multiple workers exchange or synchronize data.
- How it is used
- Synchronizes data and gradients across devices during distributed training.
- How deep it goes
- Defines broadcast and allreduce with examples, while treating the wider collective set only as a brief extension.
- Examples
- Demonstrates rank-zero broadcast of 123.0 and rank-based allreduce summation using torch.distributed.
- What you can do after
- Apply it via torch.distributed to broadcast tensors and reduce shared values across processes.
Read how Deep Learning with PyTorch, Second Edition explains collective communication →
Related concepts
- allreduceuses · Allreduce is a collective communication operation that combines corresponding values from all participating processes or devices and distributes the resulting aggregate to every participant.
- trainingused by · Model training is the process of fitting a machine-learning model to data by adjusting its parameters or structure to perform a task.
- parameter server patternalternative to · The parameter server pattern is a distributed-training pattern in which dedicated servers store and update partitions of a model's parameters.
- workeruses · A worker is a person or computational execution unit assigned to perform a task as part of a larger system or activity.
- point-to-point communicationcontrasts with · Point-to-point communication is a direct communication arrangement between two endpoints, in which a transmission is intended for one recipient rather than a group.
- communication overheadaddresses · Communication overhead is the additional time, resource consumption, and coordination effort incurred when components exchange information or synchronize their work.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.