In data-parallel training, each GPU holds a copy of the model and processes a different batch. All-reduce combines the replicas’ gradients and makes the result available to every GPU, keeping their models synchronized. section 6.3
DS2 training
Deep Speech 2 (DS2) divided training batches across 8–16 GPUs, then averaged gradients into a synchronized update. Its custom all-reduce routine enabled direct GPU-to-GPU transfers and made one full training pass 2.5 times faster than the standard OpenMPI setup. Adding GPUs roughly halved training time, but communication overhead brought diminishing returns beyond eight devices. section 4.4.2
All-reduce helps data parallelism scale while the full model fits on each device. It does not remove that memory limit: when a model outgrows one GPU, replicating it becomes infeasible, and model parallelism offers a different approach. section 6.3
