MPI (Message Passing Interface) is a standard that defines the syntax and semantics of library routines for parallel programs in C and C++. In distributed deep learning, MPI also names a backend for training across multiple machines. section 16.1.1 section 16.2
MPI-based distributed training organizes computation around processes that communicate tensor data. A world contains every process; its world size counts those processes, and each process has a unique rank. A node is one physical or virtual machine, while a process group identifies processes that can communicate. Across multiple machines, global rank identifies a process in the whole system and local rank identifies it within one node. section 16.1.1
MPI-backed distributed training commonly relies on collective communication:
- Broadcast sends data from one source process to every process in the group. section 16.2
- Allreduce combines values from every process with an operation such as sum or maximum, then distributes the result to all processes. section 16.2
- Communication performance matters because time spent exchanging gradients, parameters, and other tensors reduces time available for computation. section 16.2
Broadcast and allreduce with PyTorch
Using torch.distributed, rank 0 can create tensor([123.0]) while other ranks create zeros, then dist.broadcast(tensor, src=0) gives every rank the source value. For allreduce, each rank can start with its rank as a tensor and call dist.all_reduce(tensor, op=dist.ReduceOp.SUM), giving every process the combined sum. section 16.2
Figure 16.1 Single GPU training (left) vs. multi-GPU training (center) vs. training with multiple machines and multiple GPUs (right)
Compare single-GPU, multi-GPU, and multi-machine training layouts to see how processes, ranks, nodes, and world size scale. section 16.1.1
