chapter ten

10 Distributed computing

 

This chapter covers

  • Mapping multi-GPU interconnects and understanding NCCL collectives
  • Implementing tensor parallelism across 8 to 16 GPUs
  • Building pipeline parallel schedules that keep every stage busy
  • Coordinating data, tensor, and pipeline parallelism within one node
  • Evaluating scaling behavior and communication tradeoffs
  • Choosing the right parallelism mix for real workloads

GPUs in one machine are not all connected the same way, and the links between them set how fast your distributed code can run. The two main ways GPUs communicate inside a server are over the PCIe bus and NVLink:

  • PCIe (Peripheral Component Interconnect Express)—This is the standard bus that connects everything on a motherboard (your GPU, network cards, and storage drives) to the CPU. It’s a general-purpose interconnect.
  • NVLink—This proprietary, point-to-point interconnect was developed by NVIDIA specifically for connecting GPUs to each other and to the CPU. It offers significantly higher bandwidth and lower latency than PCIe.

NVLink 4 gives each H100 900 GB/s of GPU-to-GPU bandwidth against the 128 GB/s of its PCIe Gen 5 interface, 7× the raw bandwidth, and measured collectives stretch the gap further: NCCL AllReduce manages roughly 8 to 12 GB/s on a PCIe-only node versus about 475 GB/s of bus bandwidth on an 8-GPU NVLink ode. The first thing to do on a new multi-GPU machine is map out these links.

10.1 Understanding the hardware

10.1.1 Lab 1: Mapping your hardware

10.1.2 Understanding NCCL’s primitives

10.1.3 Ring vs. tree algorithms

10.2 Tensor parallelism

10.2.1 Initialization and GPU assignment

10.2.2 Memory allocation

10.2.3 Local matrix multiplication

10.2.4 Combining the results with AllReduce

10.3 Pipeline parallelism

10.3.1 Naive pipeline parallelism implementation: Sequential processing

10.3.3 Why streams work

10.4 Evaluating scaling strategies

10.4.1 Parallelism strategies