Model parallelism
Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
Designing Deep Learning Systems: A guide for software engineers
Model parallelism is a distributed training strategy that partitions a model's computation or parameters across multiple devices.
- How it is used
- Enables training neural networks whose full model exceeds a single device’s memory limit.
- How deep it goes
- Defines network partitioning, illustrates explicit PyTorch device transfers, and emphasizes architecture-specific implementation and utilization limits.
- Examples
- Demonstrates a four-layer network split across four GPUs and a two-GPU PyTorch subnetwork implementation.
- What you can do after
- Apply it by partitioning a large network into device-specific subnetworks and transferring intermediate outputs between GPUs.
Deep Learning with Python, Third Edition
A distributed-training method where different parts of one model are placed on different devices to fit or execute a larger model.
- How it is used
- Enables training models too large for one device by distributing their variables and computation across devices.
- How deep it goes
- Works through a dense-layer split and a Keras device-mesh configuration, including variable layouts and sharding inspection.
- Examples
- Uses a large two-layer Keras model, a two-GPU kernel split, and an eight-GPU mesh with data and model axes.
- What you can do after
- Apply it via Keras ModelParallel and LayoutMap to partition model variables across a device mesh.
Read how Deep Learning with Python, Third Edition explains model parallelism →
Sutskever's List
Model parallelism is a technique that partitions a neural network across multiple computational devices instead of replicating the whole model on each device.
- How it is used
- Addresses models too large for one device by distributing their storage and computation across devices.
- How deep it goes
- Explains partitions and communication trade-offs through AlexNet and later systems, without specifying a complete implementation procedure.
- Examples
- Uses AlexNet’s two-GPU split and Megatron-LM’s partitioned matrix multiplications as concrete designs.
- What you can do after
- Evaluate/understand when splitting a model can overcome device-memory limits and what communication or idle-time costs may follow.
Deep Learning with R, Third Edition
Model parallelism is a distributed-training strategy that partitions a model across devices so different devices execute different model components.
- How it is used
- Enables training models too large for one device by distributing their variables and computation across devices.
- How deep it goes
- Explains the device split and shows mesh, variable-layout, and strategy configuration; stops short of deriving distributed computation internals.
- Examples
- Uses an eight-GPU, two-by-four mesh and Keras LayoutMap and ModelParallel listings to illustrate combined parallelism.
- What you can do after
- Apply it via Keras ModelParallel by defining a DeviceMesh and LayoutMap, then setting the distribution strategy.
Read how Deep Learning with R, Third Edition explains model parallelism →
Deep Learning with PyTorch, Second Edition
A strategy that places different parts of a model on different devices.
- How it is used
- Enables training when a model exceeds one GPU's capacity by distributing its parameters and computations across devices.
- How deep it goes
- Works through activation transfers between three linear partitions, while treating pipeline scheduling and combined strategies separately.
- Examples
- Uses three linear model parts passing activations between ranks and a two-by-four device mesh pairing replicas with partitions.
- What you can do after
- Build a basic three-part forward pass using torch.distributed send() and recv() to transfer activations between processes.
Read how Deep Learning with PyTorch, Second Edition explains model parallelism →
Related concepts
- distributed trainingkind of · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
- pipeline parallelismincludes · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
- data parallelcontrasts with · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
- fully sharded data parallelused by · Fully Sharded Data Parallel is a distributed training technique that partitions model parameters, gradients, and optimizer states across data-parallel devices to reduce per-device memory use.
- AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
- GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and data-parallel workloads.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.