Pipeline parallelism addresses models too large for one GPU by splitting contiguous groups of Transformer blocks into stages on different GPUs. Activations pass from stage to stage, like products on an assembly line. Processing multiple micro-batches lets stages work concurrently, but sequential dependencies leave GPUs idle while the pipeline fills and drains. section 6.5.2 section 6.6
The schedule determines how much work GPUs do while data moves through the stages:
- In naive GPipe, micro-batches first move forward through the stages, then backward. The resulting pipeline bubble is idle GPU time during fill and drain. section 6.5.3
- The 1F1B schedule interleaves forward and backward passes to reduce the bubble and activation-memory cost. For MoE models, All-to-All communication can still limit efficiency. section 6.5.8
- DualPipe uses a folded, V-shaped schedule to overlap computation with MoE communication, at the cost of twice the parameter memory per GPU. section 6.5.8
Figure 6.4 The forward pass in a naive pipeline schedule. Each GPU must wait for the previous one to complete its work on a micro-batch. The white area to the left of the bars represents the initial idle time (the "bubble") as the pipeline fills up.

The forward-pass timeline shows how each stage waits for data as the pipeline fills, making the initial bubble visible. section 6.5.3





