Distributed training
Distributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.
Distributed Machine Learning Patterns
The process of fitting a machine learning model by coordinating computation across multiple machines or execution units.
- How it is used
- Positions distributed training as the central model-fitting component after data ingestion in a distributed machine learning system.
- How deep it goes
- Explains coordination patterns, bottlenecks, failures, and a TensorFlow multi-worker implementation through deployment and workflow execution.
- Examples
- Uses YouTube-8M corruption and network failures, Fashion-MNIST models, TensorFlow strategies, TFJobs, and parallel CNN variants.
- What you can do after
- Apply it via TensorFlow and Kubeflow to run synchronous multi-worker training, persist outputs, and orchestrate model-selection workflows.
Read how Distributed Machine Learning Patterns explains distributed training →
Designing Deep Learning Systems: A guide for software engineers
Distributed training is the process of training a machine learning model across multiple computing devices or processes, often with data partitioned among workers.
- How it is used
- Accelerates large-model training by coordinating multi-device worker groups through a training service.
- How deep it goes
- Explains parallelism trade-offs, worker configuration, service orchestration, and Kubernetes execution without mathematical derivations.
- Examples
- Uses TensorFlow tf_config worker addresses and indexes, an intent-classification PyTorch workload, and Kubernetes job-tracker architecture.
- What you can do after
- Operate/configure distributed training runs by selecting workers, frameworks, and Kubernetes-backed execution while preserving a unified API.
Deep Learning with Python, Third Edition
A training approach that spreads computation and/or data across multiple devices to speed up model training.
- How it is used
- Uses multi-GPU distribution to address single-device training speed and model capacity limits.
- How deep it goes
- Explains parallelism choices and Keras configuration, but defers cloud VM setup instructions.
- Examples
- Uses a 128-sample batch split across two GPUs and an eight-device mesh with dense-layer layouts.
- What you can do after
- Apply it via Keras with the JAX backend by configuring data parallelism or a model-parallel layout.
Read how Deep Learning with Python, Third Edition explains distributed training →
Deep Learning with R, Third Edition
Distributed training is the execution of model training across multiple processors or machines that coordinate computation and parameter updates.
- How it is used
- Uses multiple GPUs to train larger models or distribute training work beyond a single device.
- How deep it goes
- Explains replica synchronization and model splitting, then configures Keras distributions while deferring cloud VM setup.
- Examples
- Illustrates a 128-example batch split across two GPUs and a model replicated across two four-device groups.
- What you can do after
- Apply it via Keras and JAX by configuring data or model parallelism across GPUs.
Read how Deep Learning with R, Third Edition explains distributed training →
Deep Learning with PyTorch, Second Edition
A training approach that distributes computation or data across devices or machines.
- How it is used
- Scales deep-learning training across GPUs or machines when single-device memory, dataset size, or training time becomes limiting.
- How deep it goes
- Explains terminology, process initialization, communication, and major parallelism strategies, then stops short of exhaustive engineering detail.
- Examples
- Uses single-, multi-GPU, and multi-machine layouts, CPU experiments, torchrun jobs, and the TorchTitan training repository.
- What you can do after
- Operate/configure PyTorch jobs with process groups, multiple processes, collective communication, and selected parallelism strategies.
Read how Deep Learning with PyTorch, Second Edition explains distributed training →
Graph Neural Networks in Action
Distributed training is a technique in which model computation and data processing are divided across multiple computing processes or devices.
- How it is used
- Scales neural-network and graph-neural-network training across multiple GPUs or CPUs when single-device training is insufficient.
- How deep it goes
- Explains DDP setup, data partitioning, gradient synchronization, process barriers, multiprocessing, and communication backends.
- Examples
- Uses a two-GPU MNIST network and a related GCN example with Amazon Products and NeighborLoader.
- What you can do after
- Operate/configure PyTorch DDP training with spawned processes, synchronized gradients, device assignment, barriers, and backend selection.
Read how Graph Neural Networks in Action explains distributed training →
Also covered in
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- linear layerA linear layer is a neural-network layer that applies a learned affine transformation to its input, typically using weights and a bias.
- allreduceAllreduce is a collective communication operation that combines corresponding values from participating processes or devices and distributes the result to every participant.
- foundation modelA foundation model is a broadly trained machine-learning model whose learned capabilities can be reused or adapted for a range of downstream tasks.
Related concepts
- data parallelismincludes · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
- model parallelismincludes · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
- pipeline parallelismincludes · Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently.
- TF_CONFIGconfigured by · TF_CONFIG is an environment-variable configuration that specifies cluster and task information for TensorFlow distributed execution.
- distribution strategykind of · A distribution strategy specifies how computation and model variables are distributed across multiple devices or machines.
- JAXused by · JAX is a Python library for high-performance array computing, with composable transformations for automatic differentiation, compilation, vectorization, and parallelization.
Questions about distributed training
What is distributed training?
Distributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.
Which Manning books explain distributed training?
Distributed Machine Learning Patterns, Designing Deep Learning Systems: A guide for software engineers, Deep Learning with Python, Third Edition, Deep Learning with R, Third Edition, Deep Learning with PyTorch, Second Edition and Graph Neural Networks in Action.
What should I know before learning distributed training?
Manning books that teach distributed training usually introduce language model, linear layer, allreduce and foundation model first.
What kinds of distributed training are there?
Kinds of distributed training include data parallelism, model parallelism and pipeline parallelism.
What is distributed training a kind of?
Distributed training is a kind of distribution strategy. A distribution strategy specifies how computation and model variables are distributed across multiple devices or machines.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





