A process group is a collection of processes that can communicate with one another during distributed training. PyTorch creates it when torch.distributed.init_process_group() runs, establishing the communication context needed to exchange tensor data such as gradients and model parameters. section 16.1.1 section 16.2
Efficient communication matters because time spent exchanging tensors reduces time available for computation. Backend choice depends on the hardware: gloo is CPU-oriented, nccl is GPU-oriented, and mpi supports distributed training across multiple machines. section 16.2
Processes in the group use collective communication operations to coordinate tensor data:
- Broadcast sends data from one source process to every process in the group. section 16.2
- Allreduce combines contributions from processes and makes the resulting value available across the group, supporting synchronized updates such as gradient communication. section 16.2 section 16.12
Broadcast and allreduce
A small PyTorch example initializes distributed communication, broadcasts a tensor from rank 0, and applies allreduce with a sum operation: dist.broadcast(tensor, src=0) and dist.all_reduce(tensor, op=dist.ReduceOp.SUM). section 16.2
Figure 16.3 Illustration of the broadcast (left) and allreduce (right) collective communication operations
Visualizes how broadcast distributes one process's tensor and how allreduce combines tensors across the process group. section 16.2
