liveBook · Manning

DualPipeV

As explained in Build a DeepSeek Model (From Scratch)

DualPipeV is a pipeline-parallel training schedule that folds an N-stage pipeline into a V-shaped execution path using a single continuous stream of micro-batches.

DualPipeV folds an N-stage pipeline into a V-shaped path, placing one early and one late stage on each of N/2 physical GPUs. A micro-batch travels down the first half, turns around on the final device, and travels up the second half. section 6.5.5 section 6.5.7

For an eight-stage model on four ranks, the schedule maps and executes stages as follows:

  • Rank 0 holds Stages 0 and 7; Rank 1 holds Stages 1 and 6; Rank 2 holds Stages 2 and 5; Rank 3 holds Stages 3 and 4. section 6.5.5
  • The downstroke sends data through Stages 0→1→2→3; the turnaround continues locally from Stage 3 to Stage 4; the upstroke sends data through Stages 4→5→6→7. section 6.5.7
  • Interleaving forward and backward passes for different micro-batches reduces the pipeline bubble while using half as many physical devices as stages. section 6.5.7

DualPipeV uses one continuous micro-batch stream, making it simpler than full bidirectional DualPipe. That simplicity recovers most, but not all, of the bidirectional schedule’s bubble reduction. Each GPU nevertheless stores two stages’ worth of layers. section 6.5.7 section 6.5.7

Figure 6.9 The V-Shape Schedule of DualPipeV. Micro-batches flow down through the first half of the stages and then up through the second half, minimizing the bubble while using only N/2 GPUs for an N-stage pipeline.

Trace the V-shaped flow visually: micro-batches descend through the first stage half, turn on the final device, and ascend through the second half. section 6.5.7

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains paired stage placement, V-shaped forward and backward flow, and efficiency trade-offs, but stops short of implementation code.
Examples
Uses an eight-stage model on four GPUs and traces its downstroke, turnaround, and upstroke.
What it assumes
Assumes familiarity with pipeline stages, physical ranks, micro-batches, forward passes, backward passes, and pipeline bubbles.
What you can do after
Apply a V-shaped pipeline schedule that trades a small amount of peak efficiency for simpler single-stream execution.
How it connects
Connects pipeline parallelism with micro-batches, 1F1B scheduling, GPipe, and bidirectional pipeline execution.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.