Model parallelism
Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
Designing Deep Learning Systems: A guide for software engineers
Model parallelism is a distributed training strategy that partitions a model's computation or parameters across multiple devices.
- How it is used
- Enables training neural networks whose full model exceeds a single device’s memory limit.
- How deep it goes
- Defines network partitioning, illustrates explicit PyTorch device transfers, and emphasizes architecture-specific implementation and utilization limits.
- Examples
- Demonstrates a four-layer network split across four GPUs and a two-GPU PyTorch subnetwork implementation.
- What you can do after
- Apply it by partitioning a large network into device-specific subnetworks and transferring intermediate outputs between GPUs.
Deep Learning with Python, Third Edition
A distributed-training method where different parts of one model are placed on different devices to fit or execute a larger model.
- How it is used
- Enables training models too large to fit on one device by distributing their variables and computation.
- How deep it goes
- Explains explicit dense-computation splitting and a mesh-based configuration combining model and data parallelism.
- Examples
- Uses a two-GPU dense computation and an eight-GPU mesh with two replicas of a four-device model.
- What you can do after
- Applies model parallelism using Keras DeviceMesh, LayoutMap, and ModelParallel with JAX.
Read how Deep Learning with Python, Third Edition explains model parallelism →
Sutskever's List
Model parallelism is a technique that partitions a neural network across multiple computational devices instead of replicating the whole model on each device.
- How it is used
- Addresses models too large for one device by distributing their storage and computation across devices.
- How deep it goes
- Explains partitions and communication trade-offs through AlexNet and later systems, without specifying a complete implementation procedure.
- Examples
- Uses AlexNet’s two-GPU split and Megatron-LM’s partitioned matrix multiplications as concrete designs.
- What you can do after
- Evaluate/understand when splitting a model can overcome device-memory limits and what communication or idle-time costs may follow.
Deep Learning with R, Third Edition
Model parallelism is a distributed-training strategy that partitions a model across devices so different devices execute different model components.
- How it is used
- Enables training models too large for one device by distributing their variables and computation.
- How deep it goes
- Explains device splitting and walks through mesh, layout-map, and distribution configuration.
- Examples
- Illustrates a large dense model, a two-GPU computation split, and a two-by-four device mesh.
- What you can do after
- Apply model parallelism via Keras distribution tools using a device mesh and variable layouts.
Read how Deep Learning with R, Third Edition explains model parallelism →
Deep Learning with PyTorch, Second Edition
A strategy that places different parts of a model on different devices.
- How it is used
- Enables training models too large for one GPU by distributing their parts across devices.
- How deep it goes
- Explains partitioned forward and backward passes and demonstrates forward activation transfers, then extends the discussion to pipelines and combined strategies.
- Examples
- Uses three linear model partitions and a 2 × 4 device mesh with model replicas and partitions.
- What you can do after
- Apply it via PyTorch's distributed send and receive operations to pass activations between model partitions.
Read how Deep Learning with PyTorch, Second Edition explains model parallelism →
Also covered in
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
- Hugging Face in Action
- Deep Learning with JAX
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- activationAn activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
- parallelismParallelism is the execution of multiple independent tasks or computational operations at the same time rather than sequentially.
- backpropagationBackpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
Related concepts
- distributed trainingkind of · Distributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.
- pipeline parallelismincludes · Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently.
- data parallelismcontrasts with · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
- AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
- fully sharded data parallelused by · Fully sharded data parallel is a distributed training strategy that partitions model parameters, gradients, and optimizer states across data-parallel workers to reduce per-worker memory use.
- GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and other data-parallel workloads.
Questions about model parallelism
What is model parallelism?
Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
Which Manning books explain model parallelism?
Designing Deep Learning Systems: A guide for software engineers, Deep Learning with Python, Third Edition, Sutskever's List, Deep Learning with R, Third Edition and Deep Learning with PyTorch, Second Edition.
What should I know before learning model parallelism?
Manning books that teach model parallelism usually introduce language model, activation, parallelism and backpropagation first.
What is the difference between model parallelism and data parallelism?
Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices. Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
What kinds of model parallelism are there?
Kinds of model parallelism include pipeline parallelism.
What is model parallelism a kind of?
Model parallelism is a kind of distributed training. Distributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




