A rank is the unique integer identifier assigned to each process participating in a distributed computation. The world size counts all participating processes, while rank identifies one process within that world. section 16.1.1
In multi-machine training, distinguish global rank from local rank. Global rank identifies a process across the entire distributed system; local rank identifies it within one machine or node. section 16.1.1
Rank becomes useful alongside the main distributed-training terms:
- World size counts the processes in the complete distributed system. section 16.1.1
- A process group collects processes that can communicate with one another. section 16.1.1
- A node is one physical or virtual machine, often hosting one or more GPUs. section 16.1.1
- Distributed training uses ranks with collective operations such as broadcast and allreduce to coordinate data and gradients. section 16.12
Reading ranks across machines
With four GPUs on one machine, the world size is 4 and each process has a distinct rank. With two machines and four GPUs each, the world size is 8: global ranks identify all eight processes, while local ranks identify positions within each machine. section 16.1.1
Figure 16.1 Single GPU training (left) vs. multi-GPU training (center) vs. training with multiple machines and multiple GPUs (right)
Compare single-GPU, multi-GPU, and multi-machine layouts to see how world size, global rank, and local rank change. section 16.1.1
