A pipeline bubble is GPU idle time in pipeline-parallel execution. During the forward pass, a stage may wait for activations from the preceding stage; during the backward pass, it may wait for gradients from the following stage. Large bubbles waste compute capacity. section 6.5.1
Splitting a batch into micro-batches lets different stages work on different inputs at once. But the pipeline must first fill, and the remaining work must eventually drain. In a naive GPipe schedule, these dependencies leave substantial idle regions around the forward and backward passes. section 6.5.1 section 6.5.3
Four-stage fill
With four GPUs and eight micro-batches, only GPU 1 works at the first time step. At the second, GPU 2 can process micro-batch 1 while GPU 1 processes micro-batch 2; GPUs 3 and 4 are still waiting. The pipeline reaches full utilization only after work propagates to every stage. section 6.5.3
Figure 6.6 The combined GPipe schedule, showing the pipeline bubble. The large block of idle time (white space) between the forward and backward passes represents a massive amount of wasted computational resources.

The combined GPipe timeline makes the idle regions visible across forward and backward execution. section 6.5.3
The 1F1B schedule interleaves forward and backward work on different micro-batches instead of completing all forward passes first. Warm-up and cool-down bubbles remain, but a longer steady state makes their impact smaller. A simple pipeline efficiency expression is M / (M + P − 1), where M is the number of micro-batches and P is the number of stages. section 6.5.4
Figure 6.7 The 1F1B interleaved schedule. Forward (blue) and backward (green) passes alternate during the steady state. Note: any small gaps visible in this rendered timeline are artifacts of the profiling resolution, not the schedule itself. In a true 1F1B, each GPU executes its remaining backward passes back-to-back during cool-down.

The timeline shows alternating forward and backward tasks; small rendered gaps are profiling artifacts, not necessarily schedule idle time. section 6.5.4
Reducing the pipeline bubble does not remove every source of idle time. Mixture-of-experts layers also require cross-GPU token exchange, which can leave GPUs waiting for communication. DualPipe uses a folded stage layout and fine-grained scheduling to overlap communication with computation. section 6.5.5 section 6.6
DualPipeV simplifies the folded schedule to one stream of micro-batches: data travels down the first half of the stages and up the second half. An N-stage pipeline uses N/2 GPUs, with two stages per GPU. This saves devices but increases parameter memory per GPU; compared with bidirectional DualPipe, it trades some peak bubble reduction for simpler scheduling. section 6.5.7 section 6.6
