liveBook · Manning

GPipe

GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.

How 3 Manning books explain it

Designing Deep Learning Systems: A guide for software engineers

A pipeline-parallel training implementation that splits batches into microbatches and accumulates their gradients before updating model parameters.

How it is used
Improves GPU utilization during model-parallel training by overlapping microbatch computations across partitioned subnetworks.
How deep it goes
Explains partitioning, forward/backward scheduling, gradient accumulation, and synchronized updates; stops at a high-level workflow.
Examples
Uses a four-subnetwork, four-GPU schedule and a transformer model split across two GPUs.
What you can do after
Build it from scratch using PyTorch model partitioning and remote-communication patterns.

Read how Designing Deep Learning Systems: A guide for software engineers explains GPipe →

Sutskever's List

GPipe is a system for training neural networks by partitioning computation into pipeline stages across devices.

How it is used
Partitions neural-network computation across devices to train larger models and improve accelerator utilization.
How deep it goes
Explains staged partitioning, micro-batch scheduling, synchronized gradient accumulation, and activation recomputation, then quantifies results on two architectures.
Examples
Compares an 82-million-parameter AmoebaNet on one 8-GB GPU with larger multi-device AmoebaNet and Transformer experiments.
What you can do after
Evaluate how micro-batches, partition count, and recomputation affect model capacity, throughput, and idle-time bubbles.

Read how Sutskever's List explains GPipe →

Build a DeepSeek Model (From Scratch)

GPipe is a pipeline-parallel training schedule that divides a batch into micro-batches and executes them through model stages with a fill-and-drain pattern.

How it is used
Organizes pipeline-parallel training by distributing micro-batches across sequential model stages.
How deep it goes
Explains forward filling, backward draining, and idle bubbles with a four-GPU timeline, but stops short of complete implementation details.
Examples
Uses eight micro-batches across four GPUs and visualizes the schedule on four NVIDIA RTX 4090 GPUs.
What you can do after
Evaluate pipeline utilization by tracing micro-batch movement and identifying fill and drain bubbles.

Read how Build a DeepSeek Model (From Scratch) explains GPipe →

Also covered in

What to know first

Related concepts

Questions about GPipe

What is GPipe?

GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.

Which Manning books explain GPipe?

Designing Deep Learning Systems: A guide for software engineers, Sutskever's List and Build a DeepSeek Model (From Scratch).

What should I know before learning GPipe?

Manning books that teach GPipe usually introduce activation and backpropagation first.

What is the difference between GPipe and DualPipeV?

GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters. DualPipeV is a pipeline-parallel training schedule that folds an N-stage pipeline into a V-shaped execution path using a single continuous stream of micro-batches.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.