liveBook · Manning

Pipeline efficiency

As explained in Build a DeepSeek Model (From Scratch)

Pipeline efficiency is the fraction of total pipeline execution time spent performing computation rather than waiting in bubbles.

Pipeline efficiency measures the fraction of total pipeline execution time spent computing instead of waiting in idle bubbles. The 1F1B schedule improves it by interleaving forward and backward passes rather than completing all forwards first. section 6.5.4

For a pipeline with micro-batches and stages, efficiency is computed as: Number of Microbatches / (Number of Microbatches + Number of Stages - 1). section 6.5.4

Interpret the schedule in three phases:

  • Warm-up fills the pipeline and creates an unavoidable initial bubble. section 6.5.4
  • Steady state alternates forward and backward work to keep GPUs busy. section 6.5.4
  • Cool-down drains the remaining gradients after forward work ends. section 6.5.4
Figure 6.7 The 1F1B interleaved schedule. Forward (blue) and backward (green) passes alternate during the steady state. Note: any small gaps visible in this rendered timeline are artifacts of the profiling resolution, not the schedule itself. In a true 1F1B, each GPU executes its remaining backward passes back-to-back during cool-down.

Use the schedule timeline to see forward and backward passes alternate densely during steady state, replacing large idle regions with computation. section 6.5.4

Efficiency = (Number of Micro batches) / (Number of Microbatches + Number of Stages - 1)

Use the equation to relate efficiency directly to the number of micro-batches and pipeline stages. section 6.5.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains the efficiency equation and 1F1B warm-up, steady-state, and cool-down phases, but stops short of implementation code.
Examples
Uses a GPU 2 timeline, alternating forward and backward tasks, and an efficiency equation involving stages and micro-batches.
What it assumes
Assumes familiarity with pipeline stages, GPUs, forward passes, backward passes, gradients, and micro-batches.
What you can do after
Evaluate pipeline schedules by calculating efficiency and identifying warm-up, steady-state, and cool-down bubbles.
How it connects
Connects pipeline efficiency to pipeline parallelism, GPipe, micro-batches, and the 1F1B schedule.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.