liveBook · Manning

Data parallelism

As explained in Build a DeepSeek Model (From Scratch)

Distributed data parallelism is a training technique that replicates a model across devices and processes different data batches on each replica with synchronized parameter updates.

Data parallelism accelerates training by replicating an identical model on multiple GPUs. Each replica processes a different mini-batch, computes forward and backward passes, and synchronizes averaged gradients so every replica applies the same update. section 6.4.3

Figure 6.2 The two primary types of parallel computing in deep learning. Data Parallelism accelerates training by splitting the data, and Model Parallelism bypasses memory limits by splitting the model itself. In modern LLM training, these are not either/or choices, but complementary techniques combined to simultaneously overcome the dual constraints of time and hardware.

Use this comparison to distinguish data parallelism, which splits data, from model parallelism, which splits the model itself. section 6.4

A practical distributed setup typically follows these steps:

  • Launch one process per GPU with torchrun and assign each process a unique rank and local device. section 6.4.6
  • Initialize the process group, wrap the model with DistributedDataParallel, and use DistributedSampler to give each process a distinct data shard. section 6.4.6
  • Let DDP intercept backward passes and manage All-Reduce gradient synchronization while leaving the surrounding training loop close to the single-GPU version. section 6.4.6
Listing 6.27 Key DDP setup steps in PyTorch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data.distributed import DistributedSampler
# 1. Initialize the process group
dist.init_process_group(backend='nccl') #A
# 2. Assign the device for the current process
local_rank = int(os.environ['LOCAL_RANK'])
torch.cuda.set_device(local_rank)
# 3. Wrap the model
model = MiniDeepSeekM().to(local_rank)
ddp_model = DDP(model, device_ids=[local_rank]) #B
# 4. Use the distributed sampler for the dataloader
train_sampler = DistributedSampler(train_dataset)
train_dataloader = DataLoader(train_dataset, batch_size=..., sampler=train_sampler) #C

This listing provides the minimal PyTorch sequence: initialize communication, select the local GPU, wrap the model, and shard the dataset. section 6.4.6

Combining DDP with gradient accumulation

Suppose eight GPUs each process a physical batch of eight samples. The global batch is 8 × 8 = 64 samples. To reach an effective batch size of 4096, accumulate gradients for 4096 / 64 = 64 steps before one synchronized update. DDP addresses throughput; gradient accumulation addresses per-device batch-memory limits. section 6.4.7

The central trade-off is memory replication: DDP reduces training time by using more hardware, but the complete model, gradients, and optimizer states must fit on every GPU. Models exceeding one GPU's memory require model parallelism instead. section 6.5 section 6.5.8

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains replica synchronization and provides a PyTorch setup, but stops short of solving models that cannot fit on one GPU.
Examples
Uses sharded mini-batches, eight-GPU batch arithmetic, torchrun commands, and a minimal DistributedDataParallel configuration.
What it assumes
Assumes a working single-GPU training loop and treats gradient accumulation as the prerequisite for combining memory and throughput scaling.
What you can do after
Build it from scratch with torchrun, process groups, local-rank device mapping, DDP, and DistributedSampler.
How it connects
Connects gradient accumulation, model parallelism, pipeline parallelism, and expert parallelism as complementary responses to training bottlenecks.

Also explained in

Compare how 12 books explain data parallelism →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.