How deeply does Deep Learning with Python, Third Edition cover data parallel?

Explains replicas and batch splitting with numbers, then gives Keras setup and performance caveats; does not detail update-merging mechanics.

Data parallel

As explained in Deep Learning with Python, Third Edition

A distributed-training method where each device holds a full replica of the model and processes different sub-batches in parallel.

Data parallelism speeds up training by giving each GPU a complete copy of the model and a different share of each batch. For example, with two GPUs, a batch of 128 becomes two sub-batches of 64, processed at the same time. Training then merges the updates from the replicas. The model must fit on each device. section 18.2.1 section 18.4

keras.distribution.set_distribution(keras.distribution.DataParallel())

With Keras and JAX, set DataParallel before building the model to use data parallelism. section 18.2.2

A few practical checks matter:

To choose particular GPUs, list the available devices and pass their names to DataParallel. section 18.2.2
Keep the global batch large enough that each GPU receives enough work. Merging updates adds overhead: reported speedups are about 2× on two GPUs, 3.8× on four and 7.3× on eight. section 18.2.2
If the model cannot fit on one device, use model parallelism to split its variables and computation; the two approaches can also be combined. section 18.2.1 section 18.4

How Deep Learning with Python, Third Edition covers it

Cover of Deep Learning with Python, Third Edition

liveBook manning.com

Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Designing Deep Learning Systems: A guide for software engineersA distributed training strategy that replicates a model across computing devices, partitions data batches among them, and combines their parameter updates.
liveBook manning.com

LLMs in ProductionData parallelism is a parallel-computing technique that processes separate portions of data concurrently with multiple copies of a model or pipeline.
liveBook manning.com

Deep Learning with R, Third EditionData parallelism is a distributed-training technique that replicates a model across devices and processes different data subsets on those devices.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA training strategy that replicates a model across devices while splitting data among them.
liveBook manning.com

Sutskever's ListData parallelism is a training technique that replicates a model across devices, splits batches among them, and synchronizes parameter updates.
liveBook manning.com

Deep Learning with JAXA distributed training strategy that splits a batch across devices, replicates parameters, and aggregates gradients before updating.
liveBook manning.com

LatencyData parallelism is the execution of identical operations simultaneously on separate portions of a dataset.

Compare how 8 books explain data parallel →

Related concepts

distributed trainingkind of · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
rayonimplemented by · Rayon is a Rust library for data-parallel programming through parallel iterators and work distribution across threads.
model parallelismcontrasts with · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and data-parallel workloads.
parallelismkind of · Parallelism is the concurrent execution of multiple computational operations, often across multiple processing resources.
allreduceuses · Allreduce is a collective communication operation that combines corresponding values across participating processes or devices and distributes the resulting aggregate to every participant.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.