liveBook · Manning

Data parallelism

Also called data parallel training

Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.

How 12 Manning books explain it

Designing Deep Learning Systems: A guide for software engineers

A distributed training strategy that replicates a model across computing devices, partitions data batches among them, and combines their parameter updates.

How it is used
Accelerates training of large datasets by distributing minibatches across replicated model workers.
How deep it goes
Explains synchronous execution, gradient aggregation, scaling limits, recovery, and framework configuration without model-theory detail.
Examples
Demonstrates three-GPU synchronization plus PyTorch DDP, TensorFlow MultiWorkerMirroredStrategy, and Horovod code patterns.
What you can do after
Operate/configure it through PyTorch, TensorFlow, or Horovod worker settings and resilient training-service practices.

Read how Designing Deep Learning Systems: A guide for software engineers explains data parallelism →

CUDA for LLMs

Data parallelism distributes different portions of input data across devices that perform corresponding computations on model replicas.

How it is used
Uses model replication and batch partitioning to speed distributed training when a model fits on one GPU.
How deep it goes
Explains the training sequence, gradient synchronization, scaling estimate, and ring AllReduce costs without providing a DistributedDataParallel implementation.
Examples
Illustrates an eight-GPU image classifier with 256 images and element-wise ReLU across GPU threads.
What you can do after
Evaluate and understand when replicated training can speed up a fitting model and when communication or batch size may limit it.

Read how CUDA for LLMs explains data parallelism →

Grokking GPU Programming

Data parallelism performs the same operation concurrently on multiple pieces of data.

How it is used
Introduces a recurring approach for distributing independent, identical work across GPU threads.
How deep it goes
Explains sequential and parallel execution, thread indexing, bounds checks, and synchronization; previews multidimensional layouts.
Examples
Illustrates the approach with array doubling and image shading.
What you can do after
Apply it via CUDA by assigning data to threads, checking bounds, and synchronizing before using results.

Read how Grokking GPU Programming explains data parallelism →

LLMs in Production

Data parallelism is a parallel-computing technique that processes separate portions of data concurrently with multiple copies of a model or pipeline.

How it is used
Uses replicated model copies to process separate data portions concurrently in distributed LLM training and inference workflows.
How deep it goes
Defines the workload split and names PyTorch configuration, but stops short of explaining synchronization or communication internals.
Examples
Demonstrates Ray task submission by comparing ten sequential calls with ten remote calls and collected results.
What you can do after
Apply it via PyTorch DistributedDataParallel or Ray when distributing compatible workloads across devices or cluster nodes.

Read how LLMs in Production explains data parallelism →

Deep Learning with Python, Third Edition

A distributed-training method where each device holds a full replica of the model and processes different sub-batches in parallel.

How it is used
Uses data parallelism to scale training across devices when a model fits on one device.
How deep it goes
Explains replica and batch splitting, result merging, device selection, and scaling overhead without detailing the merge algorithm.
Examples
Illustrates a 128-sample batch split across two GPUs and configures two named GPUs.
What you can do after
Apply data parallelism via Keras DataParallel with JAX, optionally selecting specific devices.

Read how Deep Learning with Python, Third Edition explains data parallelism →

Deep Learning with R, Third Edition

Data parallelism is a distributed-training technique that replicates a model across devices and processes different data subsets on those devices.

How it is used
Uses data parallelism to scale training across multiple devices when a model fits on one device.
How deep it goes
Explains batch splitting, gradient averaging, inference, configuration, and scaling overhead, while deferring cloud VM setup.
Examples
Illustrates a batch of 128 split across two GPUs and configures DataParallel with default or explicitly selected devices.
What you can do after
Apply data parallelism via Keras DataParallel with JAX, optionally selecting devices and considering batch size.

Read how Deep Learning with R, Third Edition explains data parallelism →

Deep Learning with PyTorch, Second Edition

A training strategy that replicates a model across devices while splitting data among them.

How it is used
Uses model replication and data splitting to scale training across devices while keeping updates consistent.
How deep it goes
Works through gradient averaging and DDP wrapping, then addresses dataset sharding with DistributedSampler.
Examples
Demonstrates manual gradient reduction, DDP model wrapping, and an MNIST data loader using DistributedSampler.
What you can do after
Apply it via PyTorch DistributedDataParallel and DistributedSampler to distribute training and data loading.

Read how Deep Learning with PyTorch, Second Edition explains data parallelism →

Sutskever's List

Data parallelism is a training technique that replicates a model across devices, splits batches among them, and synchronizes parameter updates.

How it is used
Scales training across GPUs by distributing batches while keeping model replicas synchronized.
How deep it goes
Explains batch division, gradient synchronization, measured speedups, communication overhead, and memory limits; does not detail the custom all-reduce routine.
Examples
Uses Deep Speech 2’s 8-to-16-GPU runs, batch sizes, and custom all-reduce performance as a concrete case.
What you can do after
Evaluate/understand when data parallelism speeds GPU training and when communication or full-model memory limits constrain it.

Read how Sutskever's List explains data parallelism →

Build a DeepSeek Model (From Scratch)

Distributed data parallelism is a training technique that replicates a model across devices and processes different data batches on each replica with synchronized parameter updates.

How it is used
Accelerates large-model training by processing separate data shards concurrently across replicated GPU models.
How deep it goes
Explains replica synchronization and provides a PyTorch setup, but stops short of solving models that cannot fit on one GPU.
Examples
Uses sharded mini-batches, eight-GPU batch arithmetic, torchrun commands, and a minimal DistributedDataParallel configuration.
What you can do after
Build it from scratch with torchrun, process groups, local-rank device mapping, DDP, and DistributedSampler.

Read how Build a DeepSeek Model (From Scratch) explains data parallelism →

Deep Learning with JAX

A distributed training strategy that splits a batch across devices, replicates parameters, and aggregates gradients before updating.

How it is used
Uses multiple devices to train one neural network by processing separate data chunks with synchronized model copies.
How deep it goes
Explains distribution, reshaping, gradient aggregation, and edge cases, while deferring model-parallel implementation to later mechanisms.
Examples
Demonstrates an SPMD MNIST classifier using eight-device tensors, pmap updates, and per-device loss values.
What you can do after
Apply it via JAX pmap by reshaping batches, aggregating gradients, and completing a multi-device training loop.

Read how Deep Learning with JAX explains data parallelism →

Latency

Data parallelism is the execution of identical operations simultaneously on separate portions of a dataset.

How it is used
Uses data parallelism to reduce latency and improve throughput by processing portions of a dataset simultaneously.
How deep it goes
Explains partitioning, workload balance, locality, and hardware capacity through a worked reduction and idealized scaling estimates.
Examples
Illustrates parallel work with an array sum, contiguous matrix blocks, and AVX-512 element counts.
What you can do after
Evaluate data-parallel opportunities by considering chunk independence, workload balance, locality, and available processing capacity.

Read how Latency explains data parallelism →

GPU Programming with Triton

Data parallelism applies the same computation independently to multiple partitions of data.

How it is used
Explains why GPUs suit workloads whose output elements can be computed independently.
How deep it goes
Defines independent-output work and its GPU motivation, but does not develop a kernel implementation.
Examples
Uses element-wise tensor arithmetic, where each output depends on one input and requires no worker coordination.
What you can do after
Understand why GPUs suit independent element-wise workloads and how Triton maps work across tiles.

Read how GPU Programming with Triton explains data parallelism →

Also covered in

What to know first

Related concepts

Questions about data parallelism

What is data parallelism?

Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.

Which Manning books explain data parallelism?

Designing Deep Learning Systems: A guide for software engineers, CUDA for LLMs, Grokking GPU Programming, LLMs in Production, Deep Learning with Python, Third Edition, Deep Learning with R, Third Edition, Deep Learning with PyTorch, Second Edition, Sutskever's List and 4 more.

What should I know before learning data parallelism?

Manning books that teach data parallelism usually introduce gradient, matrix-matrix multiplication, throughput and parameter first.

What is the difference between data parallelism and model parallelism?

Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data. Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.

What kinds of data parallelism are there?

Kinds of data parallelism include fully sharded data parallel.

What is data parallelism a kind of?

Data parallelism is a kind of distributed training. Distributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.