Pipeline parallelism
Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
Sutskever's List
Pipeline parallelism is a distributed-training technique that splits a model into sequential stages and executes microbatches through those stages concurrently.
- How it is used
- Uses GPipe to train models beyond one accelerator’s memory while increasing throughput through overlapping micro-batches.
- How deep it goes
- Explains stages, scheduling, synchronized updates, checkpointing, and measured scaling, but stops short of a runnable implementation.
- Examples
- Compares AmoebaNet and Transformer capacity and speed measurements, plus earlier eight-GPU sequence-to-sequence pipelining.
- What you can do after
- Evaluate/understand it as a capacity-and-throughput strategy with memory savings, idle-time bubbles, and implementation costs.
Designing Deep Learning Systems: A guide for software engineers
A distributed training technique that partitions a model into sequential stages executed by different workers, allowing batches to flow through the stages concurrently.
- How it is used
- Trains models too large for one GPU by distributing sequential network stages across multiple devices.
- How deep it goes
- Explains microbatch overlap, model partitioning, PyTorch pipeline setup, and operational support, while deferring any general model-splitting method.
- Examples
- Uses partitioned sequential networks, a GPU-utilization timeline, and a two-GPU transformer pipeline with remote communication.
- What you can do after
- Operate/configure it through a training service that allocates workers, enables communication, distributes parameters, and supports recovery planning.
LLMs in Production
Pipeline parallelism is a distributed-computing technique that partitions a model into sequential stages placed on different devices.
- How it is used
- Makes models that exceed a single GPU’s memory runnable by distributing sequential model stages across devices.
- How deep it goes
- Explains vertical partitioning, device transfers, idle bubbles, and microbatching, but does not detail scheduling algorithms.
- Examples
- Uses a sequential multi-GPU flow and a bubble-versus-microbatching comparison to demonstrate execution and utilization.
- What you can do after
- Evaluate when pipeline parallelism is necessary and diagnose underutilization caused by pipeline bubbles.
Deep Learning with PyTorch, Second Edition
A model-parallel technique that splits a model into stages and overlaps their execution.
- How it is used
- Scales training and inference for large models by distributing sequential stages across devices and overlapping microbatch execution.
- How deep it goes
- Explains stages, microbatches, tradeoffs, architecture fit, and PyTorch setup, but defers detailed schedule mechanics and performance tuning.
- Examples
- Uses a three-device microbatch diagram and a PyTorch MLP with split points, stages, and rank-specific schedule steps.
- What you can do after
- Build it from scratch with torch.distributed.pipelining by marking split points, constructing stages, and running a pipeline schedule.
Read how Deep Learning with PyTorch, Second Edition explains pipeline parallelism →
Related concepts
- model parallelismkind of · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
- GPipeimplemented by · GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.
- batchinguses · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.
- distributed trainingkind of · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
- tensor parallelismcontrasts with · Tensor parallelism is a model-parallel technique that partitions tensor computations and their associated parameters across multiple devices and combines their partial results.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.