liveBook · Manning

Collective communication

Also called collective operation

Collective communication is a coordinated operation in which multiple processes or devices exchange, combine, or synchronize data as part of a distributed computation.

How 3 Manning books explain it

Distributed Machine Learning Patterns

A collective communication pattern coordinates communication of computation updates among multiple workers and synchronizes their model copies.

How it is used
Speeds distributed training of small or medium-sized models when parameter-server communication becomes a bottleneck.
How deep it goes
Explains worker-only architecture, reduce, broadcast, allreduce, and ring-allreduce, while deferring implementation details.
Examples
Uses contending workers and parameter servers, plus a 30 MB Fashion-MNIST classification model with worker-only allreduce.
What you can do after
Evaluate/understand when worker-only allreduce can replace parameter servers for distributed model training.

Read how Distributed Machine Learning Patterns explains collective communication →

CUDA for LLMs

A collective operation is a coordinated communication operation involving a group of distributed processes or devices.

How it is used
Coordinates GPU data movement and combines partial results in distributed training and tensor-parallel computation.
How deep it goes
Defines major communication patterns and explains topology and scaling tradeoffs, while leaving collective kernel implementations to NCCL.
Examples
Uses four-GPU gradient summation and an eight-GPU matrix multiply with measured AllReduce overhead.
What you can do after
Evaluate collective communication costs using topology, message size, bandwidth benchmarks, and end-to-end workload measurements.

Read how CUDA for LLMs explains collective communication →

Deep Learning with PyTorch, Second Edition

A distributed communication pattern in which multiple workers exchange or synchronize data.

How it is used
Synchronizes data and gradients across devices during distributed training.
How deep it goes
Defines broadcast and allreduce with examples, while treating the wider collective set only as a brief extension.
Examples
Demonstrates rank-zero broadcast of 123.0 and rank-based allreduce summation using torch.distributed.
What you can do after
Apply it via torch.distributed to broadcast tensors and reduce shared values across processes.

Read how Deep Learning with PyTorch, Second Edition explains collective communication →

Also covered in

What to know first

Related concepts

Questions about collective communication

What is collective communication?

Collective communication is a coordinated operation in which multiple processes or devices exchange, combine, or synchronize data as part of a distributed computation.

Which Manning books explain collective communication?

Distributed Machine Learning Patterns, CUDA for LLMs and Deep Learning with PyTorch, Second Edition.

What should I know before learning collective communication?

Manning books that teach collective communication usually introduce gradient, parameter, parallelism and data parallelism first.

What is the difference between collective communication and parameter server pattern?

Collective communication is a coordinated operation in which multiple processes or devices exchange, combine, or synchronize data as part of a distributed computation. The parameter server pattern is a distributed-training pattern in which dedicated servers store and update partitions of a model's parameters.

What kinds of collective communication are there?

Kinds of collective communication include allreduce and reduction operation.

What is collective communication also called?

Collective communication is also called collective operation.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.