GPipe
GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.
Designing Deep Learning Systems: A guide for software engineers
A pipeline-parallel training implementation that splits batches into microbatches and accumulates their gradients before updating model parameters.
- How it is used
- Improves GPU utilization during model-parallel training by overlapping microbatch computations across partitioned subnetworks.
- How deep it goes
- Explains partitioning, forward/backward scheduling, gradient accumulation, and synchronized updates; stops at a high-level workflow.
- Examples
- Uses a four-subnetwork, four-GPU schedule and a transformer model split across two GPUs.
- What you can do after
- Build it from scratch using PyTorch model partitioning and remote-communication patterns.
Read how Designing Deep Learning Systems: A guide for software engineers explains GPipe →
Sutskever's List
GPipe is a system for training neural networks by partitioning computation into pipeline stages across devices.
- How it is used
- Partitions neural-network computation across devices to train larger models and improve accelerator utilization.
- How deep it goes
- Explains staged partitioning, micro-batch scheduling, synchronized gradient accumulation, and activation recomputation, then quantifies results on two architectures.
- Examples
- Compares an 82-million-parameter AmoebaNet on one 8-GB GPU with larger multi-device AmoebaNet and Transformer experiments.
- What you can do after
- Evaluate how micro-batches, partition count, and recomputation affect model capacity, throughput, and idle-time bubbles.
Related concepts
- pipeline parallelismimplements · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
- AmoebaNetuses · AmoebaNet is a convolutional neural network architecture discovered through neural architecture search for image recognition.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.