Pipeline parallelism

As explained in Sutskever's List

Pipeline parallelism is a distributed-training technique that splits a model into sequential stages and executes microbatches through those stages concurrently.

Splitting a neural network’s layers across accelerators solves a memory problem but creates a scheduling problem: in a simple sequential setup, most devices wait while one works. GPipe’s pipeline parallelism divides the model into stages, then splits each training batch into micro-batches. Different stages can work on different micro-batches at the same time, improving accelerator utilization without requiring every device to hold the full model. section 6.3.1

Figure 6.6 Pipeline parallelism compared to naive model parallelism. (a) A simple neural network partitioned across four accelerators, with forward passes F₀ – F₃ flowing upward and backward passes B₀ – B₃ flowing downward to compute gradients. (b) In naive model parallelism, layers are split sequentially across devices. Because each stage must wait for the previous stage to complete, much of the hardware remains idle, leading to low utilization. (c) Pipeline parallelism mitigates this inefficiency by splitting each mini-batch into smaller micro-batches. These micro-batches flow through the pipeline in a staggered fashion, keeping devices busy by processing different micro-batches simultaneously. While idle-time bubbles still appear during the pipeline “fill” and “drain” phases, overall utilization improves significantly. Gradients are accumulated across micro-batches, and updates are applied synchronously at the end of each full batch. Used with permission granted by the lead author (Yanping Huang).

The four-accelerator comparison shows where a sequential split leaves devices idle and how staggered micro-batches reduce that idle time. It also shows the remaining gaps while the pipeline fills and drains. section 6.3.1

GPipe accumulates gradients across the micro-batches and makes one synchronized parameter update at the end of the full batch, avoiding updates based on stale weights. It also saves memory through activation checkpointing: selected intermediate values are recomputed during backpropagation rather than stored throughout the pass. That exchanges extra computation for memory capacity. Using at least as many micro-batches as partitions can make pipeline idle time small, though it does not remove every overhead. section 6.3.1 section 6.3.2

Capacity and speed in GPipe experiments

An 8 GB GPU could train an AmoebaNet of about 82 million parameters on its own; partitioning with GPipe across eight GPUs raised that limit to 1.8 billion. For Transformers, 128 TPU partitions supported an 83.9-billion-parameter model, versus approximately 282 million parameters on one 16 GB device. An eight-partition Transformer pipeline ran 6.3 times faster than a single partition—substantial, but short of the ideal eightfold gain. section 6.3.2

The trade-off extends beyond raw speed. Pipeline code must coordinate micro-batch schedules, communication, and forward and backward passes; devices still wait during pipeline fill and drain. Looking back in 2024, Ilya Sutskever questioned whether earlier pipelining had been wise despite its memory and speed gains. Megatron-LM offered a different split—partitioning computations *within* layers—and that tensor-parallel approach can also be combined with pipeline stages. The practical choice is therefore not simply whether pipelining works, but whether its capacity gains justify its scheduling and implementation costs. section 6.3.3

How Sutskever's List covers it

How deep it goes

Explains stages, scheduling, synchronized updates, checkpointing, and measured scaling, but stops short of a runnable implementation.

Examples

Compares AmoebaNet and Transformer capacity and speed measurements, plus earlier eight-GPU sequence-to-sequence pipelining.

What it assumes

Introduces micro-batches and activation checkpointing while relying on basic familiarity with training batches, gradients, and accelerators.

What you can do after

Evaluate/understand it as a capacity-and-throughput strategy with memory savings, idle-time bubbles, and implementation costs.

How it connects

Relates pipeline parallelism to model parallelism, activation checkpointing, tensor parallelism, and data parallelism.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Designing Deep Learning Systems: A guide for software engineersA distributed training technique that partitions a model into sequential stages executed by different workers, allowing batches to flow through the stages concurrently.
• 

  liveBookmanning.com
  LLMs in ProductionPipeline parallelism is a distributed-computing technique that partitions a model into sequential stages placed on different devices.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA model-parallel technique that splits a model into stages and overlaps their execution.

Compare how 4 books explain pipeline parallelism →

Related concepts

• model parallelismkind of · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
• GPipeimplemented by · GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.
• batchinguses · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.
• distributed trainingkind of · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
• tensor parallelismcontrasts with · Tensor parallelism is a model-parallel technique that partitions tensor computations and their associated parameters across multiple devices and combines their partial results.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.