Synchronous data parallelism reduces training time by replicating a model across CPU or GPU workers. Each worker processes a different minibatch, gradients are aggregated, and every replica updates with the same result. section 4.2.1
A synchronous training iteration adds two coordination steps to single-device training:
- Divide one training batch into minibatches and distribute them across the workers. section 4.2.1
- Run the training steps simultaneously using separate data partitions. section 4.2.1
- Aggregate gradients, commonly with all-reduce, and dispatch the result to every worker before local model updates. section 4.2.1
Tracking a distributed training job
A Kubernetes-based training service can query the master pod to represent overall progress because synchronous workers must synchronize during every training cycle. The tracker moves jobs between launching, running, and finalized queues as pod status changes. section 4.3.4
Figure 4.1 A synchronous data parallelism concept graph. (a) Deep learning training on a single GPU. (b) Synchronous data parallel training with three GPUs.
Compare single-GPU training with a three-GPU synchronous setup, including minibatch splitting and gradient synchronization. section 4.2.1
