GPipe
GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.
Designing Deep Learning Systems: A guide for software engineers
A pipeline-parallel training implementation that splits batches into microbatches and accumulates their gradients before updating model parameters.
- How it is used
- Improves GPU utilization during model-parallel training by overlapping microbatch computations across partitioned subnetworks.
- How deep it goes
- Explains partitioning, forward/backward scheduling, gradient accumulation, and synchronized updates; stops at a high-level workflow.
- Examples
- Uses a four-subnetwork, four-GPU schedule and a transformer model split across two GPUs.
- What you can do after
- Build it from scratch using PyTorch model partitioning and remote-communication patterns.
Read how Designing Deep Learning Systems: A guide for software engineers explains GPipe →
Sutskever's List
GPipe is a system for training neural networks by partitioning computation into pipeline stages across devices.
- How it is used
- Partitions neural-network computation across devices to train larger models and improve accelerator utilization.
- How deep it goes
- Explains staged partitioning, micro-batch scheduling, synchronized gradient accumulation, and activation recomputation, then quantifies results on two architectures.
- Examples
- Compares an 82-million-parameter AmoebaNet on one 8-GB GPU with larger multi-device AmoebaNet and Transformer experiments.
- What you can do after
- Evaluate how micro-batches, partition count, and recomputation affect model capacity, throughput, and idle-time bubbles.
Build a DeepSeek Model (From Scratch)
GPipe is a pipeline-parallel training schedule that divides a batch into micro-batches and executes them through model stages with a fill-and-drain pattern.
- How it is used
- Organizes pipeline-parallel training by distributing micro-batches across sequential model stages.
- How deep it goes
- Explains forward filling, backward draining, and idle bubbles with a four-GPU timeline, but stops short of complete implementation details.
- Examples
- Uses eight micro-batches across four GPUs and visualizes the schedule on four NVIDIA RTX 4090 GPUs.
- What you can do after
- Evaluate pipeline utilization by tracing micro-batch movement and identifying fill and drain bubbles.
Read how Build a DeepSeek Model (From Scratch) explains GPipe →
Also covered in
What to know first
- activationAn activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
- backpropagationBackpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
Related concepts
- pipeline parallelismimplements · Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently.
- gradient checkpointinguses · Gradient checkpointing is a memory-saving training technique that stores selected intermediate activations and recomputes others during backpropagation.
- DualPipeVcontrasts with · DualPipeV is a pipeline-parallel training schedule that folds an N-stage pipeline into a V-shaped execution path using a single continuous stream of micro-batches.
- pipeline bubbleproduces · A pipeline bubble is an interval or empty slot in pipelined computation during which one or more processing stages are idle because work is unavailable or discarded.
- 1F1B schedulealternative to · A 1F1B schedule is a pipeline-parallel execution schedule that interleaves one forward micro-batch operation with one backward micro-batch operation.
- AmoebaNetuses · AmoebaNet is a convolutional neural network architecture discovered through neural architecture search for image recognition.
Questions about GPipe
What is GPipe?
GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.
Which Manning books explain GPipe?
Designing Deep Learning Systems: A guide for software engineers, Sutskever's List and Build a DeepSeek Model (From Scratch).
What should I know before learning GPipe?
Manning books that teach GPipe usually introduce activation and backpropagation first.
What is the difference between GPipe and DualPipeV?
GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters. DualPipeV is a pipeline-parallel training schedule that folds an N-stage pipeline into a V-shaped execution path using a single continuous stream of micro-batches.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


