Distributed training
Distributed training is the coordinated training of a machine-learning model across multiple devices, processes, or machines by dividing computation or data among them.
Distributed Machine Learning Patterns
The process of fitting a machine learning model by coordinating computation across multiple machines or execution units.
- How it is used
- Positions distributed training as the central model-fitting component after data ingestion in a distributed machine learning system.
- How deep it goes
- Explains coordination patterns, bottlenecks, failures, and a TensorFlow multi-worker implementation through deployment and workflow execution.
- Examples
- Uses YouTube-8M corruption and network failures, Fashion-MNIST models, TensorFlow strategies, TFJobs, and parallel CNN variants.
- What you can do after
- Apply it via TensorFlow and Kubeflow to run synchronous multi-worker training, persist outputs, and orchestrate model-selection workflows.
Read how Distributed Machine Learning Patterns explains distributed training →
Designing Deep Learning Systems: A guide for software engineers
Distributed training is the process of training a machine learning model across multiple computing devices or processes, often with data partitioned among workers.
- How it is used
- Accelerates large-model training by coordinating multi-device worker groups through a training service.
- How deep it goes
- Explains parallelism trade-offs, worker configuration, service orchestration, and Kubernetes execution without mathematical derivations.
- Examples
- Uses TensorFlow tf_config worker addresses and indexes, an intent-classification PyTorch workload, and Kubernetes job-tracker architecture.
- What you can do after
- Operate/configure distributed training runs by selecting workers, frameworks, and Kubernetes-backed execution while preserving a unified API.
Deep Learning with Python, Third Edition
A training approach that spreads computation and/or data across multiple devices to speed up model training.
- How it is used
- Accelerates training beyond one GPU by distributing batches or model components across devices.
- How deep it goes
- Works through replicas, batch splitting, scaling limits, and JAX mesh configuration; leaves cloud VM setup instructions aside.
- Examples
- Uses a batch of 128 on two GPUs, an eight-device combined layout, and Keras distribution listings.
- What you can do after
- Apply it via Keras and JAX by setting DataParallel or configuring a ModelParallel device mesh and layout map.
Read how Deep Learning with Python, Third Edition explains distributed training →
Deep Learning with R, Third Edition
Distributed training is the execution of model training across multiple processors or machines that coordinate computation and parameter updates.
- How it is used
- Uses multiple GPUs to accelerate training or accommodate models too large for one device.
- How deep it goes
- Explains data and model parallelism with batch-splitting examples and Keras configuration, but defers cloud VM setup.
- Examples
- Illustrates a 128-sample batch across two GPUs and a two-by-four device mesh.
- What you can do after
- Apply it via Keras and JAX to configure data parallelism and specify a combined data-and-model device layout.
Read how Deep Learning with R, Third Edition explains distributed training →
Deep Learning with PyTorch, Second Edition
A training approach that distributes computation or data across devices or machines.
- How it is used
- Scales deep-learning training across GPUs or machines when single-device memory, dataset size, or training time becomes limiting.
- How deep it goes
- Explains terminology, process initialization, communication, and major parallelism strategies, then stops short of exhaustive engineering detail.
- Examples
- Uses single-, multi-GPU, and multi-machine layouts, CPU experiments, torchrun jobs, and the TorchTitan training repository.
- What you can do after
- Operate/configure PyTorch jobs with process groups, multiple processes, collective communication, and selected parallelism strategies.
Read how Deep Learning with PyTorch, Second Edition explains distributed training →
Graph Neural Networks in Action
Distributed training is a technique in which model computation and data processing are divided across multiple computing processes or devices.
- How it is used
- Scales neural-network and graph-neural-network training across multiple GPUs or CPUs when single-device training is insufficient.
- How deep it goes
- Explains DDP setup, data partitioning, gradient synchronization, process barriers, multiprocessing, and communication backends.
- Examples
- Uses a two-GPU MNIST network and a related GCN example with Amazon Products and NeighborLoader.
- What you can do after
- Operate/configure PyTorch DDP training with spawned processes, synchronized gradients, device assignment, barriers, and backend selection.
Read how Graph Neural Networks in Action explains distributed training →
Related concepts
- data parallelincludes · Data parallelism is a parallel-computing technique that applies the same computation to separate portions of data concurrently, often by replicating the computation across workers.
- model parallelismincludes · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
- pipeline parallelismincludes · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
- TF_CONFIGconfigured by · TF_CONFIG is an environment-variable configuration that specifies cluster and task information for TensorFlow distributed execution.
- Kubeflowused by · Kubeflow is an open-source platform for developing, deploying, and orchestrating machine-learning workflows on Kubernetes.
- JAXused by · JAX is a Python library for high-performance array-based numerical computing with composable transformations including automatic differentiation, compilation, vectorization, and parallelization.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.