DualPipeV folds an N-stage pipeline into a V-shaped path, placing one early and one late stage on each of N/2 physical GPUs. A micro-batch travels down the first half, turns around on the final device, and travels up the second half. section 6.5.5 section 6.5.7
For an eight-stage model on four ranks, the schedule maps and executes stages as follows:
- Rank 0 holds Stages 0 and 7; Rank 1 holds Stages 1 and 6; Rank 2 holds Stages 2 and 5; Rank 3 holds Stages 3 and 4. section 6.5.5
- The downstroke sends data through Stages 0→1→2→3; the turnaround continues locally from Stage 3 to Stage 4; the upstroke sends data through Stages 4→5→6→7. section 6.5.7
- Interleaving forward and backward passes for different micro-batches reduces the pipeline bubble while using half as many physical devices as stages. section 6.5.7
DualPipeV uses one continuous micro-batch stream, making it simpler than full bidirectional DualPipe. That simplicity recovers most, but not all, of the bidirectional schedule’s bubble reduction. Each GPU nevertheless stores two stages’ worth of layers. section 6.5.7 section 6.5.7
Figure 6.9 The V-Shape Schedule of DualPipeV. Micro-batches flow down through the first half of the stages and then up through the second half, minimizing the bubble while using only N/2 GPUs for an N-stage pipeline.

Trace the V-shaped flow visually: micro-batches descend through the first stage half, turn on the final device, and ascend through the second half. section 6.5.7
