liveBook · Manning

DistributedDataParallel

Also called DDP

DistributedDataParallel is a PyTorch API that trains replicated models across processes or devices by synchronizing gradients during data-parallel computation.

How 3 Manning books explain it

Graph Neural Networks in Action

DistributedDataParallel is a PyTorch API that runs replicated model processes across multiple GPUs or machines while synchronizing gradients.

How it is used
Uses DistributedDataParallel to spread training across multiple GPUs, with small-scale clusters also considered.
How deep it goes
Walks through process groups, manual data division, model wrapping, training, synchronization, and cleanup, then illustrates a two-GPU script.
Examples
Demonstrates MNIST training with a Flatten-and-Linear network and points to a GCN example using Amazon Products.
What you can do after
Apply it via PyTorch DistributedDataParallel to run multi-GPU training while dividing data and coordinating processes.

Read how Graph Neural Networks in Action explains DistributedDataParallel →

Designing Deep Learning Systems: A guide for software engineers

PyTorch's DistributedDataParallel library component that runs a model across processes or machines and coordinates distributed updates.

How it is used
Runs PyTorch data-parallel training across processes on one machine or across multiple machines.
How deep it goes
Explains process registration, model wrapping, gradient synchronization, and bucket tuning without descending into custom synchronization internals.
Examples
Uses process-group setup, GPU model wrapping, synchronized backward, and gradient-bucket sizing as implementation examples.
What you can do after
Apply it via PyTorch by initializing a process group, wrapping a model with DDP, and training with synchronized gradients.

Read how Designing Deep Learning Systems: A guide for software engineers explains DistributedDataParallel →

Build a DeepSeek Model (From Scratch)

nn.DistributedDataParallel is a PyTorch module for synchronously training replicated models across multiple processes or devices.

How it is used
Scales synchronous model training across GPUs by combining local computation with peer-to-peer gradient averaging.
How deep it goes
Explains process setup, data sharding, backward-pass all-reduce, communication overlap, and synchronized optimizer updates.
Examples
Uses a four-GPU torchrun command, a batch-sharding diagram, and a PyTorch setup listing.
What you can do after
Apply via PyTorch by launching processes, assigning devices, wrapping models with DDP, and partitioning data with DistributedSampler.

Read how Build a DeepSeek Model (From Scratch) explains DistributedDataParallel →

Also covered in

What to know first

Related concepts

Questions about DistributedDataParallel

What is DistributedDataParallel?

DistributedDataParallel is a PyTorch API that trains replicated models across processes or devices by synchronizing gradients during data-parallel computation.

Which Manning books explain DistributedDataParallel?

Graph Neural Networks in Action, Designing Deep Learning Systems: A guide for software engineers and Build a DeepSeek Model (From Scratch).

What should I know before learning DistributedDataParallel?

Manning books that teach DistributedDataParallel usually introduce gradient, backpropagation, allreduce and collective communication first.

What is the difference between DistributedDataParallel and DataParallel?

DistributedDataParallel is a PyTorch API that trains replicated models across processes or devices by synchronizing gradients during data-parallel computation. DataParallel is a parallel-computation mechanism that replicates model computation across devices while distributing input data among them.

What is DistributedDataParallel also called?

DistributedDataParallel is also called DDP.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.