liveBook · Manning

Distributed training

Distributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.

How 6 Manning books explain it

Distributed Machine Learning Patterns

The process of fitting a machine learning model by coordinating computation across multiple machines or execution units.

How it is used
Positions distributed training as the central model-fitting component after data ingestion in a distributed machine learning system.
How deep it goes
Explains coordination patterns, bottlenecks, failures, and a TensorFlow multi-worker implementation through deployment and workflow execution.
Examples
Uses YouTube-8M corruption and network failures, Fashion-MNIST models, TensorFlow strategies, TFJobs, and parallel CNN variants.
What you can do after
Apply it via TensorFlow and Kubeflow to run synchronous multi-worker training, persist outputs, and orchestrate model-selection workflows.

Read how Distributed Machine Learning Patterns explains distributed training →

Designing Deep Learning Systems: A guide for software engineers

Distributed training is the process of training a machine learning model across multiple computing devices or processes, often with data partitioned among workers.

How it is used
Accelerates large-model training by coordinating multi-device worker groups through a training service.
How deep it goes
Explains parallelism trade-offs, worker configuration, service orchestration, and Kubernetes execution without mathematical derivations.
Examples
Uses TensorFlow tf_config worker addresses and indexes, an intent-classification PyTorch workload, and Kubernetes job-tracker architecture.
What you can do after
Operate/configure distributed training runs by selecting workers, frameworks, and Kubernetes-backed execution while preserving a unified API.

Read how Designing Deep Learning Systems: A guide for software engineers explains distributed training →

Deep Learning with Python, Third Edition

A training approach that spreads computation and/or data across multiple devices to speed up model training.

How it is used
Uses multi-GPU distribution to address single-device training speed and model capacity limits.
How deep it goes
Explains parallelism choices and Keras configuration, but defers cloud VM setup instructions.
Examples
Uses a 128-sample batch split across two GPUs and an eight-device mesh with dense-layer layouts.
What you can do after
Apply it via Keras with the JAX backend by configuring data parallelism or a model-parallel layout.

Read how Deep Learning with Python, Third Edition explains distributed training →

Deep Learning with R, Third Edition

Distributed training is the execution of model training across multiple processors or machines that coordinate computation and parameter updates.

How it is used
Uses multiple GPUs to train larger models or distribute training work beyond a single device.
How deep it goes
Explains replica synchronization and model splitting, then configures Keras distributions while deferring cloud VM setup.
Examples
Illustrates a 128-example batch split across two GPUs and a model replicated across two four-device groups.
What you can do after
Apply it via Keras and JAX by configuring data or model parallelism across GPUs.

Read how Deep Learning with R, Third Edition explains distributed training →

Deep Learning with PyTorch, Second Edition

A training approach that distributes computation or data across devices or machines.

How it is used
Scales deep-learning training across GPUs or machines when single-device memory, dataset size, or training time becomes limiting.
How deep it goes
Explains terminology, process initialization, communication, and major parallelism strategies, then stops short of exhaustive engineering detail.
Examples
Uses single-, multi-GPU, and multi-machine layouts, CPU experiments, torchrun jobs, and the TorchTitan training repository.
What you can do after
Operate/configure PyTorch jobs with process groups, multiple processes, collective communication, and selected parallelism strategies.

Read how Deep Learning with PyTorch, Second Edition explains distributed training →

Graph Neural Networks in Action

Distributed training is a technique in which model computation and data processing are divided across multiple computing processes or devices.

How it is used
Scales neural-network and graph-neural-network training across multiple GPUs or CPUs when single-device training is insufficient.
How deep it goes
Explains DDP setup, data partitioning, gradient synchronization, process barriers, multiprocessing, and communication backends.
Examples
Uses a two-GPU MNIST network and a related GCN example with Amazon Products and NeighborLoader.
What you can do after
Operate/configure PyTorch DDP training with spawned processes, synchronized gradients, device assignment, barriers, and backend selection.

Read how Graph Neural Networks in Action explains distributed training →

Also covered in

What to know first

Related concepts

Questions about distributed training

What is distributed training?

Distributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.

Which Manning books explain distributed training?

Distributed Machine Learning Patterns, Designing Deep Learning Systems: A guide for software engineers, Deep Learning with Python, Third Edition, Deep Learning with R, Third Edition, Deep Learning with PyTorch, Second Edition and Graph Neural Networks in Action.

What should I know before learning distributed training?

Manning books that teach distributed training usually introduce language model, linear layer, allreduce and foundation model first.

What kinds of distributed training are there?

Kinds of distributed training include data parallelism, model parallelism and pipeline parallelism.

What is distributed training a kind of?

Distributed training is a kind of distribution strategy. A distribution strategy specifies how computation and model variables are distributed across multiple devices or machines.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.