Data parallelism
Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
Designing Deep Learning Systems: A guide for software engineers
A distributed training strategy that replicates a model across computing devices, partitions data batches among them, and combines their parameter updates.
- How it is used
- Accelerates training of large datasets by distributing minibatches across replicated model workers.
- How deep it goes
- Explains synchronous execution, gradient aggregation, scaling limits, recovery, and framework configuration without model-theory detail.
- Examples
- Demonstrates three-GPU synchronization plus PyTorch DDP, TensorFlow MultiWorkerMirroredStrategy, and Horovod code patterns.
- What you can do after
- Operate/configure it through PyTorch, TensorFlow, or Horovod worker settings and resilient training-service practices.
Read how Designing Deep Learning Systems: A guide for software engineers explains data parallelism →
CUDA for LLMs
Data parallelism distributes different portions of input data across devices that perform corresponding computations on model replicas.
- How it is used
- Uses model replication and batch partitioning to speed distributed training when a model fits on one GPU.
- How deep it goes
- Explains the training sequence, gradient synchronization, scaling estimate, and ring AllReduce costs without providing a DistributedDataParallel implementation.
- Examples
- Illustrates an eight-GPU image classifier with 256 images and element-wise ReLU across GPU threads.
- What you can do after
- Evaluate and understand when replicated training can speed up a fitting model and when communication or batch size may limit it.
Grokking GPU Programming
Data parallelism performs the same operation concurrently on multiple pieces of data.
- How it is used
- Introduces a recurring approach for distributing independent, identical work across GPU threads.
- How deep it goes
- Explains sequential and parallel execution, thread indexing, bounds checks, and synchronization; previews multidimensional layouts.
- Examples
- Illustrates the approach with array doubling and image shading.
- What you can do after
- Apply it via CUDA by assigning data to threads, checking bounds, and synchronizing before using results.
Read how Grokking GPU Programming explains data parallelism →
LLMs in Production
Data parallelism is a parallel-computing technique that processes separate portions of data concurrently with multiple copies of a model or pipeline.
- How it is used
- Uses replicated model copies to process separate data portions concurrently in distributed LLM training and inference workflows.
- How deep it goes
- Defines the workload split and names PyTorch configuration, but stops short of explaining synchronization or communication internals.
- Examples
- Demonstrates Ray task submission by comparing ten sequential calls with ten remote calls and collected results.
- What you can do after
- Apply it via PyTorch DistributedDataParallel or Ray when distributing compatible workloads across devices or cluster nodes.
Deep Learning with Python, Third Edition
A distributed-training method where each device holds a full replica of the model and processes different sub-batches in parallel.
- How it is used
- Uses data parallelism to scale training across devices when a model fits on one device.
- How deep it goes
- Explains replica and batch splitting, result merging, device selection, and scaling overhead without detailing the merge algorithm.
- Examples
- Illustrates a 128-sample batch split across two GPUs and configures two named GPUs.
- What you can do after
- Apply data parallelism via Keras DataParallel with JAX, optionally selecting specific devices.
Read how Deep Learning with Python, Third Edition explains data parallelism →
Deep Learning with R, Third Edition
Data parallelism is a distributed-training technique that replicates a model across devices and processes different data subsets on those devices.
- How it is used
- Uses data parallelism to scale training across multiple devices when a model fits on one device.
- How deep it goes
- Explains batch splitting, gradient averaging, inference, configuration, and scaling overhead, while deferring cloud VM setup.
- Examples
- Illustrates a batch of 128 split across two GPUs and configures DataParallel with default or explicitly selected devices.
- What you can do after
- Apply data parallelism via Keras DataParallel with JAX, optionally selecting devices and considering batch size.
Read how Deep Learning with R, Third Edition explains data parallelism →
Deep Learning with PyTorch, Second Edition
A training strategy that replicates a model across devices while splitting data among them.
- How it is used
- Uses model replication and data splitting to scale training across devices while keeping updates consistent.
- How deep it goes
- Works through gradient averaging and DDP wrapping, then addresses dataset sharding with DistributedSampler.
- Examples
- Demonstrates manual gradient reduction, DDP model wrapping, and an MNIST data loader using DistributedSampler.
- What you can do after
- Apply it via PyTorch DistributedDataParallel and DistributedSampler to distribute training and data loading.
Read how Deep Learning with PyTorch, Second Edition explains data parallelism →
Sutskever's List
Data parallelism is a training technique that replicates a model across devices, splits batches among them, and synchronizes parameter updates.
- How it is used
- Scales training across GPUs by distributing batches while keeping model replicas synchronized.
- How deep it goes
- Explains batch division, gradient synchronization, measured speedups, communication overhead, and memory limits; does not detail the custom all-reduce routine.
- Examples
- Uses Deep Speech 2’s 8-to-16-GPU runs, batch sizes, and custom all-reduce performance as a concrete case.
- What you can do after
- Evaluate/understand when data parallelism speeds GPU training and when communication or full-model memory limits constrain it.
Build a DeepSeek Model (From Scratch)
Distributed data parallelism is a training technique that replicates a model across devices and processes different data batches on each replica with synchronized parameter updates.
- How it is used
- Accelerates large-model training by processing separate data shards concurrently across replicated GPU models.
- How deep it goes
- Explains replica synchronization and provides a PyTorch setup, but stops short of solving models that cannot fit on one GPU.
- Examples
- Uses sharded mini-batches, eight-GPU batch arithmetic, torchrun commands, and a minimal DistributedDataParallel configuration.
- What you can do after
- Build it from scratch with torchrun, process groups, local-rank device mapping, DDP, and DistributedSampler.
Read how Build a DeepSeek Model (From Scratch) explains data parallelism →
Deep Learning with JAX
A distributed training strategy that splits a batch across devices, replicates parameters, and aggregates gradients before updating.
- How it is used
- Uses multiple devices to train one neural network by processing separate data chunks with synchronized model copies.
- How deep it goes
- Explains distribution, reshaping, gradient aggregation, and edge cases, while deferring model-parallel implementation to later mechanisms.
- Examples
- Demonstrates an SPMD MNIST classifier using eight-device tensors, pmap updates, and per-device loss values.
- What you can do after
- Apply it via JAX pmap by reshaping batches, aggregating gradients, and completing a multi-device training loop.
Latency
Data parallelism is the execution of identical operations simultaneously on separate portions of a dataset.
- How it is used
- Uses data parallelism to reduce latency and improve throughput by processing portions of a dataset simultaneously.
- How deep it goes
- Explains partitioning, workload balance, locality, and hardware capacity through a worked reduction and idealized scaling estimates.
- Examples
- Illustrates parallel work with an array sum, contiguous matrix blocks, and AVX-512 element counts.
- What you can do after
- Evaluate data-parallel opportunities by considering chunk independence, workload balance, locality, and available processing capacity.
GPU Programming with Triton
Data parallelism applies the same computation independently to multiple partitions of data.
- How it is used
- Explains why GPUs suit workloads whose output elements can be computed independently.
- How deep it goes
- Defines independent-output work and its GPU motivation, but does not develop a kernel implementation.
- Examples
- Uses element-wise tensor arithmetic, where each output depends on one input and requires no worker coordination.
- What you can do after
- Understand why GPUs suit independent element-wise workloads and how Triton maps work across tiles.
Read how GPU Programming with Triton explains data parallelism →
Also covered in
- C++ Concurrency in Action, Second Edition
- Rust in Action: Systems programming concepts and techniques
What to know first
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- throughputThroughput is the rate at which a system processes or completes work, data, or requests over a specified period.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
Related concepts
- distributed trainingkind of · Distributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.
- fully sharded data parallelincludes · Fully sharded data parallel is a distributed training strategy that partitions model parameters, gradients, and optimizer states across data-parallel workers to reduce per-worker memory use.
- rayonimplemented by · Rayon is a Rust library for data-parallel programming through parallel iterators and work distribution across threads.
- GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and other data-parallel workloads.
- model parallelismcontrasts with · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
Questions about data parallelism
What is data parallelism?
Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
Which Manning books explain data parallelism?
Designing Deep Learning Systems: A guide for software engineers, CUDA for LLMs, Grokking GPU Programming, LLMs in Production, Deep Learning with Python, Third Edition, Deep Learning with R, Third Edition, Deep Learning with PyTorch, Second Edition, Sutskever's List and 4 more.
What should I know before learning data parallelism?
Manning books that teach data parallelism usually introduce gradient, matrix-matrix multiplication, throughput and parameter first.
What is the difference between data parallelism and model parallelism?
Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data. Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
What kinds of data parallelism are there?
Kinds of data parallelism include fully sharded data parallel.
What is data parallelism a kind of?
Data parallelism is a kind of distributed training. Distributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.











