A larger training run does not always require splitting the model itself. With data parallelism, every GPU holds a copy of the full network, but processes a different share of the training batch. The GPUs then combine their gradients through an all-reduce operation so their model copies remain synchronized. section 6.3
Deep Speech 2 across GPUs
Deep Speech 2 typically used 8 to 16 GPUs and batches of 512 to 1,024 samples—64 samples per GPU. After each iteration, gradients were averaged into one synchronized update. A custom all-reduce routine using direct GPU-to-GPU transfers made a full training pass 2.5 times faster than an off-the-shelf OpenMPI setup. Doubling the GPU count roughly halved training time, but communication overhead brought diminishing returns beyond eight devices. section 4.4.2
The trade-off is memory. Because each GPU stores the entire model, data parallelism stops being feasible when the model exceeds one device’s capacity. Model or tensor parallelism instead splits parts of a model across devices; Mesh-TensorFlow expresses data parallelism as splitting the batch axis. ZeRO addresses a different cost by eliminating duplicate storage across data-parallel processes. section 6.3 section 6.3.3
