GPipe trains a staged model by splitting a large batch into micro-batches and sending them through the stages sequentially. The pipeline fills during the forward pass, then drains during the reverse-direction backward pass. section 6.5.3
A naive GPipe schedule exposes two sources of idle time:
- During filling, later GPUs wait for activations from earlier stages; with four GPUs, they initially wait for one, two, and three time steps. section 6.5.3
- During draining, backward computation proceeds from the last GPU toward the first, leaving idle time as the pipeline empties. section 6.5.3
- The resulting pipeline bubble represents wasted computational capacity between or around active work. section 6.5.3
Figure 6.6 The combined GPipe schedule, showing the pipeline bubble. The large block of idle time (white space) between the forward and backward passes represents a massive amount of wasted computational resources.

The combined schedule diagram makes the fill, backward drain, and intervening pipeline bubble visible across model stages. section 6.5.3


