liveBook · Manning

Pipeline parallelism

As explained in Build a DeepSeek Model (From Scratch)

Pipeline parallelism is a model-parallel training technique that divides computation into sequential stages and processes multiple micro-batches through them.

Pipeline parallelism addresses models too large for one GPU by splitting contiguous groups of Transformer blocks into stages on different GPUs. Activations pass from stage to stage, like products on an assembly line. Processing multiple micro-batches lets stages work concurrently, but sequential dependencies leave GPUs idle while the pipeline fills and drains. section 6.5.2 section 6.6

The schedule determines how much work GPUs do while data moves through the stages:

  • In naive GPipe, micro-batches first move forward through the stages, then backward. The resulting pipeline bubble is idle GPU time during fill and drain. section 6.5.3
  • The 1F1B schedule interleaves forward and backward passes to reduce the bubble and activation-memory cost. For MoE models, All-to-All communication can still limit efficiency. section 6.5.8
  • DualPipe uses a folded, V-shaped schedule to overlap computation with MoE communication, at the cost of twice the parameter memory per GPU. section 6.5.8
Figure 6.4 The forward pass in a naive pipeline schedule. Each GPU must wait for the previous one to complete its work on a micro-batch. The white area to the left of the bars represents the initial idle time (the "bubble") as the pipeline fills up.

The forward-pass timeline shows how each stage waits for data as the pipeline fills, making the initial bubble visible. section 6.5.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains stage placement, micro-batch flow, schedule bubbles, and trade-offs across GPipe, 1F1B, and DualPipe.
Examples
Illustrates GPipe with eight micro-batches across four GPUs and a hands-on four-GPU visualization.
What it assumes
Defines micro-batches, stages, communication, and pipeline bubbles before comparing schedules.
What you can do after
Understand how stage dependencies create pipeline bubbles and how alternative schedules trade idle time against memory and communication costs.
How it connects
Connects model parallelism with Transformer blocks, GPipe, 1F1B, MoE, and All-to-All communication.

Also explained in

Compare how 6 books explain pipeline parallelism →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.