liveBook · Manning

DistributedDataParallel

As explained in Build a DeepSeek Model (From Scratch)

nn.DistributedDataParallel is a PyTorch module for synchronously training replicated models across multiple processes or devices.

DistributedDataParallel (DDP) trains replicated PyTorch models synchronously across processes or GPUs. Each process owns an identical model and optimizer, consumes a distinct data shard, computes locally, and participates in gradient averaging so every replica applies the same update. Unlike centralized DataParallel, DDP removes the coordinating GPU and scales through peer-to-peer communication. section 6.4.3 section 6.4.5

Figure 6.3 The workflow of Data Parallelism. A large data batch is sharded into mini-batches, and each GPU processes one mini-batch using its local copy of the model.

Visualizes batch sharding, replicated models, local computation, and cross-GPU gradient aggregation. section 6.4.3

A practical DDP setup follows these steps:

  • Launch one process per GPU, commonly with torchrun, and assign each process a rank and local device. section 6.4.6
  • Initialize the distributed process group to establish communication among participating processes. section 6.4.6
  • Wrap the device-local model with DistributedDataParallel so backward passes trigger gradient synchronization. section 6.4.6
  • Use DistributedSampler to give each process a unique dataset shard while retaining an otherwise familiar training loop. section 6.4.6

Four-process launch

torchrun --nproc_per_node=4 train.py --args launches train.py in four processes, one for each available GPU.

Listing 6.27 Key DDP setup steps in PyTorch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data.distributed import DistributedSampler
# 1. Initialize the process group
dist.init_process_group(backend='nccl') #A
# 2. Assign the device for the current process
local_rank = int(os.environ['LOCAL_RANK'])
torch.cuda.set_device(local_rank)
# 3. Wrap the model
model = MiniDeepSeekM().to(local_rank)
ddp_model = DDP(model, device_ids=[local_rank]) #B
# 4. Use the distributed sampler for the dataloader
train_sampler = DistributedSampler(train_dataset)
train_dataloader = DataLoader(train_dataset, batch_size=..., sampler=train_sampler) #C

Provides the compact PyTorch initialization pattern: process group, local device, DDP wrapper, and distributed sampler. section 6.4.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains process setup, data sharding, backward-pass all-reduce, communication overlap, and synchronized optimizer updates.
Examples
Uses a four-GPU torchrun command, a batch-sharding diagram, and a PyTorch setup listing.
What it assumes
Assumes familiarity with PyTorch models, optimizers, GPUs, forward and backward passes, and ordinary single-GPU training loops.
What you can do after
Apply via PyTorch by launching processes, assigning devices, wrapping models with DDP, and partitioning data with DistributedSampler.
How it connects
Connects data parallelism, gradient synchronization, all-reduce, DistributedSampler, torchrun, and PyTorch training loops.

Also explained in

Compare how 3 books explain DistributedDataParallel →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.