Designed initially for real-time video-game graphics, GPUs proved well suited to the repeated computations used to train artificial neural networks. AlexNet made that fit consequential: in 2012, its team used two GPUs to train a network too large for either device’s memory alone. section 2.5.4
GPU training can be scaled in different ways, each with its own communication costs:
- AlexNet used model parallelism: each GPU handled half the filters and feature maps. The team synchronized at selected layers to share information while limiting communication overhead. section 2.5.4
- Deep Speech 2 used data parallelism: GPUs processed separate portions of each batch, then averaged gradients for a synchronized update. Communication overhead reduced the gains from adding devices beyond eight. section 4.4.2
Scaling Deep Speech 2
A custom all-reduce routine enabled direct GPU-to-GPU transfers and made one full training pass 2.5 times faster than the standard OpenMPI setup. Doubling the GPU count roughly halved training time, until communication overhead began to erode the benefit. section 4.4.2
