liveBook · Manning

GPipe

As explained in Build a DeepSeek Model (From Scratch)

GPipe is a pipeline-parallel training schedule that divides a batch into micro-batches and executes them through model stages with a fill-and-drain pattern.

GPipe trains a staged model by splitting a large batch into micro-batches and sending them through the stages sequentially. The pipeline fills during the forward pass, then drains during the reverse-direction backward pass. section 6.5.3

A naive GPipe schedule exposes two sources of idle time:

  • During filling, later GPUs wait for activations from earlier stages; with four GPUs, they initially wait for one, two, and three time steps. section 6.5.3
  • During draining, backward computation proceeds from the last GPU toward the first, leaving idle time as the pipeline empties. section 6.5.3
  • The resulting pipeline bubble represents wasted computational capacity between or around active work. section 6.5.3
Figure 6.6 The combined GPipe schedule, showing the pipeline bubble. The large block of idle time (white space) between the forward and backward passes represents a massive amount of wasted computational resources.

The combined schedule diagram makes the fill, backward drain, and intervening pipeline bubble visible across model stages. section 6.5.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains forward filling, backward draining, and idle bubbles with a four-GPU timeline, but stops short of complete implementation details.
Examples
Uses eight micro-batches across four GPUs and visualizes the schedule on four NVIDIA RTX 4090 GPUs.
What it assumes
Assumes familiarity with model stages, GPUs, batches, activations, and forward and backward computation.
What you can do after
Evaluate pipeline utilization by tracing micro-batch movement and identifying fill and drain bubbles.
How it connects
Connects micro-batches, model stages, activations, forward pass, backward pass, and pipeline bubbles.

Also explained in

Compare how 3 books explain GPipe →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.