liveBook · Manning

Pipeline parallelism

Also called pipelining

Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently.

How 6 Manning books explain it

Sutskever's List

Pipeline parallelism is a distributed-training technique that splits a model into sequential stages and executes microbatches through those stages concurrently.

How it is used
Uses GPipe to train models beyond one accelerator’s memory while increasing throughput through overlapping micro-batches.
How deep it goes
Explains stages, scheduling, synchronized updates, checkpointing, and measured scaling, but stops short of a runnable implementation.
Examples
Compares AmoebaNet and Transformer capacity and speed measurements, plus earlier eight-GPU sequence-to-sequence pipelining.
What you can do after
Evaluate/understand it as a capacity-and-throughput strategy with memory savings, idle-time bubbles, and implementation costs.

Read how Sutskever's List explains pipeline parallelism →

Designing Deep Learning Systems: A guide for software engineers

A distributed training technique that partitions a model into sequential stages executed by different workers, allowing batches to flow through the stages concurrently.

How it is used
Trains models too large for one GPU by distributing sequential network stages across multiple devices.
How deep it goes
Explains microbatch overlap, model partitioning, PyTorch pipeline setup, and operational support, while deferring any general model-splitting method.
Examples
Uses partitioned sequential networks, a GPU-utilization timeline, and a two-GPU transformer pipeline with remote communication.
What you can do after
Operate/configure it through a training service that allocates workers, enables communication, distributes parameters, and supports recovery planning.

Read how Designing Deep Learning Systems: A guide for software engineers explains pipeline parallelism →

LLMs in Production

Pipeline parallelism is a distributed-computing technique that partitions a model into sequential stages placed on different devices.

How it is used
Enables models too large for one GPU to run by placing sequential model stages on different GPUs.
How deep it goes
Explains model partitioning, sequential transfers, utilization bubbles, and microbatching as a mitigation.
Examples
Contrasts processing all data at once with sending smaller batches through model stages.
What you can do after
Understand how staged execution can make an oversized model runnable and how smaller batches reduce GPU idle time.

Read how LLMs in Production explains pipeline parallelism →

CUDA for LLMs

Pipeline parallelism partitions a model into sequential stages assigned to different devices so that multiple inputs can be processed concurrently.

How it is used
Distributes deep models across GPUs and overlaps batches to improve throughput.
How deep it goes
Traces a sequential CUDA pipeline and contrasts blocking synchronization with stream-based overlap, stage balance, and latency tradeoffs.
Examples
Uses a four-GPU MLP with 4096-wide hidden layers and compares sequential and stream-based batch processing.
What you can do after
Apply it via CUDA streams to overlap batches, and evaluate stage balance, throughput, latency, and transfer costs.

Read how CUDA for LLMs explains pipeline parallelism →

Build a DeepSeek Model (From Scratch)

Pipeline parallelism is a model-parallel training technique that divides computation into sequential stages and processes multiple micro-batches through them.

How it is used
Solves the model-size bottleneck by distributing model layers across GPUs.
How deep it goes
Explains stage placement, micro-batch flow, schedule bubbles, and trade-offs across GPipe, 1F1B, and DualPipe.
Examples
Illustrates GPipe with eight micro-batches across four GPUs and a hands-on four-GPU visualization.
What you can do after
Understand how stage dependencies create pipeline bubbles and how alternative schedules trade idle time against memory and communication costs.

Read how Build a DeepSeek Model (From Scratch) explains pipeline parallelism →

Deep Learning with PyTorch, Second Edition

A model-parallel technique that splits a model into stages and overlaps their execution.

How it is used
Scales training and inference for large language models by overlapping work across devices.
How deep it goes
Explains stages, microbatch overlap, scheduling utilities, and balancing challenges without detailing runtime internals.
Examples
Uses a three-device pipeline, an MLP with layer split points, and deep-versus-wide model comparisons.
What you can do after
Apply it with PyTorch's `torch.distributed.pipelining` utilities to split models, build stages, and execute schedules.

Read how Deep Learning with PyTorch, Second Edition explains pipeline parallelism →

Also covered in

What to know first

Related concepts

Questions about pipeline parallelism

What is pipeline parallelism?

Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently.

Which Manning books explain pipeline parallelism?

Sutskever's List, Designing Deep Learning Systems: A guide for software engineers, LLMs in Production, CUDA for LLMs, Build a DeepSeek Model (From Scratch) and Deep Learning with PyTorch, Second Edition.

What should I know before learning pipeline parallelism?

Manning books that teach pipeline parallelism usually introduce activation, language model, data parallelism and distributed training first.

What is the difference between pipeline parallelism and tensor parallelism?

Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently. Tensor parallelism is a model-parallel technique that partitions tensor computations or associated parameters across multiple devices and combines their partial results.

What is pipeline parallelism a kind of?

Pipeline parallelism is a kind of model parallelism. Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.

What is pipeline parallelism also called?

Pipeline parallelism is also called pipelining.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.