chapter ten
10 Distributed computing
This chapter covers
- Mapping multi-GPU interconnects and understanding NCCL collectives
- Implementing tensor parallelism across 8 to 16 GPUs
- Building pipeline parallel schedules that keep every stage busy
- Coordinating data, tensor, and pipeline parallelism within one node
- Evaluating scaling behavior and communication tradeoffs
- Choosing the right parallelism mix for real workloads
GPUs in one machine are not all connected the same way, and the links between them set how fast your distributed code can run. The two main ways GPUs communicate inside a server are over the PCIe bus and NVLink:
- PCIe (Peripheral Component Interconnect Express)—This is the standard bus that connects everything on a motherboard (your GPU, network cards, and storage drives) to the CPU. It’s a general-purpose interconnect.
- NVLink—This proprietary, point-to-point interconnect was developed by NVIDIA specifically for connecting GPUs to each other and to the CPU. It offers significantly higher bandwidth and lower latency than PCIe.
NVLink 4 gives each H100 900 GB/s of GPU-to-GPU bandwidth against the 128 GB/s of its PCIe Gen 5 interface, 7× the raw bandwidth, and measured collectives stretch the gap further: NCCL AllReduce manages roughly 8 to 12 GB/s on a PCIe-only node versus about 475 GB/s of bus bandwidth on an 8-GPU NVLink ode. The first thing to do on a new multi-GPU machine is map out these links.