Pipeline parallelism
Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently.
Sutskever's List
Pipeline parallelism is a distributed-training technique that splits a model into sequential stages and executes microbatches through those stages concurrently.
- How it is used
- Uses GPipe to train models beyond one accelerator’s memory while increasing throughput through overlapping micro-batches.
- How deep it goes
- Explains stages, scheduling, synchronized updates, checkpointing, and measured scaling, but stops short of a runnable implementation.
- Examples
- Compares AmoebaNet and Transformer capacity and speed measurements, plus earlier eight-GPU sequence-to-sequence pipelining.
- What you can do after
- Evaluate/understand it as a capacity-and-throughput strategy with memory savings, idle-time bubbles, and implementation costs.
Designing Deep Learning Systems: A guide for software engineers
A distributed training technique that partitions a model into sequential stages executed by different workers, allowing batches to flow through the stages concurrently.
- How it is used
- Trains models too large for one GPU by distributing sequential network stages across multiple devices.
- How deep it goes
- Explains microbatch overlap, model partitioning, PyTorch pipeline setup, and operational support, while deferring any general model-splitting method.
- Examples
- Uses partitioned sequential networks, a GPU-utilization timeline, and a two-GPU transformer pipeline with remote communication.
- What you can do after
- Operate/configure it through a training service that allocates workers, enables communication, distributes parameters, and supports recovery planning.
LLMs in Production
Pipeline parallelism is a distributed-computing technique that partitions a model into sequential stages placed on different devices.
- How it is used
- Enables models too large for one GPU to run by placing sequential model stages on different GPUs.
- How deep it goes
- Explains model partitioning, sequential transfers, utilization bubbles, and microbatching as a mitigation.
- Examples
- Contrasts processing all data at once with sending smaller batches through model stages.
- What you can do after
- Understand how staged execution can make an oversized model runnable and how smaller batches reduce GPU idle time.
CUDA for LLMs
Pipeline parallelism partitions a model into sequential stages assigned to different devices so that multiple inputs can be processed concurrently.
- How it is used
- Distributes deep models across GPUs and overlaps batches to improve throughput.
- How deep it goes
- Traces a sequential CUDA pipeline and contrasts blocking synchronization with stream-based overlap, stage balance, and latency tradeoffs.
- Examples
- Uses a four-GPU MLP with 4096-wide hidden layers and compares sequential and stream-based batch processing.
- What you can do after
- Apply it via CUDA streams to overlap batches, and evaluate stage balance, throughput, latency, and transfer costs.
Build a DeepSeek Model (From Scratch)
Pipeline parallelism is a model-parallel training technique that divides computation into sequential stages and processes multiple micro-batches through them.
- How it is used
- Solves the model-size bottleneck by distributing model layers across GPUs.
- How deep it goes
- Explains stage placement, micro-batch flow, schedule bubbles, and trade-offs across GPipe, 1F1B, and DualPipe.
- Examples
- Illustrates GPipe with eight micro-batches across four GPUs and a hands-on four-GPU visualization.
- What you can do after
- Understand how stage dependencies create pipeline bubbles and how alternative schedules trade idle time against memory and communication costs.
Read how Build a DeepSeek Model (From Scratch) explains pipeline parallelism →
Deep Learning with PyTorch, Second Edition
A model-parallel technique that splits a model into stages and overlaps their execution.
- How it is used
- Scales training and inference for large language models by overlapping work across devices.
- How deep it goes
- Explains stages, microbatch overlap, scheduling utilities, and balancing challenges without detailing runtime internals.
- Examples
- Uses a three-device pipeline, an MLP with layer split points, and deep-versus-wide model comparisons.
- What you can do after
- Apply it with PyTorch's `torch.distributed.pipelining` utilities to split models, build stages, and execute schedules.
Read how Deep Learning with PyTorch, Second Edition explains pipeline parallelism →
Also covered in
What to know first
- activationAn activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- data parallelismData parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
- distributed trainingDistributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.
Related concepts
- batchinguses · Batching is the grouping of multiple items, requests, or operations so they can be processed together as a unit.
- GPipeimplemented by · GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.
- model parallelismkind of · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
- tensor parallelismcontrasts with · Tensor parallelism is a model-parallel technique that partitions tensor computations or associated parameters across multiple devices and combines their partial results.
- distributed trainingkind of · Distributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.
- concurrencyuses · Concurrency is the structuring or execution of multiple computational activities so their progress overlaps in time, whether or not they run simultaneously.
Questions about pipeline parallelism
What is pipeline parallelism?
Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently.
Which Manning books explain pipeline parallelism?
Sutskever's List, Designing Deep Learning Systems: A guide for software engineers, LLMs in Production, CUDA for LLMs, Build a DeepSeek Model (From Scratch) and Deep Learning with PyTorch, Second Edition.
What should I know before learning pipeline parallelism?
Manning books that teach pipeline parallelism usually introduce activation, language model, data parallelism and distributed training first.
What is the difference between pipeline parallelism and tensor parallelism?
Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently. Tensor parallelism is a model-parallel technique that partitions tensor computations or associated parameters across multiple devices and combines their partial results.
What is pipeline parallelism a kind of?
Pipeline parallelism is a kind of model parallelism. Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
What is pipeline parallelism also called?
Pipeline parallelism is also called pipelining.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





