Gradient synchronization
Gradient synchronization is the coordination of gradients computed by distributed training processes so replicated models receive consistent parameter updates.
Designing Deep Learning Systems: A guide for software engineers
The coordination of gradient values across distributed training workers so their model updates remain consistent.
- How it is used
- Keeps data-parallel workers aligned while they train separate minibatches and update a shared model.
- How deep it goes
- Explains synchronous and asynchronous behavior, worker registration, framework wrappers, and lower-level synchronization APIs.
- Examples
- Uses PyTorch DDP, TensorFlow MultiWorkerMirroredStrategy, and Horovod gradient tapes or distributed optimizers.
- What you can do after
- Apply it via PyTorch DDP, TensorFlow MultiWorkerMirroredStrategy, or Horovod distributed training APIs.
Graph Neural Networks in Action
Gradient synchronization is the coordination of gradients computed by multiple training processes so that replicated models receive consistent parameter updates.
- How it is used
- Coordinates distributed training updates when multiple GPU processes train replicated models on separate data subsets.
- How deep it goes
- Explains process setup, device assignment, model wrapping, backward passes, and synchronized updates without deriving synchronization internals.
- Examples
- Uses MNIST with a simple neural network and mentions a GCN Amazon Products variant on a dual-GPU Kaggle notebook.
- What you can do after
- Apply it via PyTorch DistributedDataParallel and multiprocessing to train across multiple GPUs.
Read how Graph Neural Networks in Action explains gradient synchronization →
Related concepts
- DistributedDataParallelused by · DistributedDataParallel is a PyTorch module that replicates a model across processes or devices and synchronizes gradients during data-parallel training.
- data parallelused by · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.