liveBook · Manning

Pipeline bubble

As explained in Build a DeepSeek Model (From Scratch)

A pipeline bubble is an interval in pipeline-parallel execution when one or more accelerator stages are idle because work has not yet reached them or has already left them.

A pipeline bubble is GPU idle time in pipeline-parallel execution. During the forward pass, a stage may wait for activations from the preceding stage; during the backward pass, it may wait for gradients from the following stage. Large bubbles waste compute capacity. section 6.5.1

Splitting a batch into micro-batches lets different stages work on different inputs at once. But the pipeline must first fill, and the remaining work must eventually drain. In a naive GPipe schedule, these dependencies leave substantial idle regions around the forward and backward passes. section 6.5.1 section 6.5.3

Four-stage fill

With four GPUs and eight micro-batches, only GPU 1 works at the first time step. At the second, GPU 2 can process micro-batch 1 while GPU 1 processes micro-batch 2; GPUs 3 and 4 are still waiting. The pipeline reaches full utilization only after work propagates to every stage. section 6.5.3

Figure 6.6 The combined GPipe schedule, showing the pipeline bubble. The large block of idle time (white space) between the forward and backward passes represents a massive amount of wasted computational resources.

The combined GPipe timeline makes the idle regions visible across forward and backward execution. section 6.5.3

The 1F1B schedule interleaves forward and backward work on different micro-batches instead of completing all forward passes first. Warm-up and cool-down bubbles remain, but a longer steady state makes their impact smaller. A simple pipeline efficiency expression is M / (M + P − 1), where M is the number of micro-batches and P is the number of stages. section 6.5.4

Figure 6.7 The 1F1B interleaved schedule. Forward (blue) and backward (green) passes alternate during the steady state. Note: any small gaps visible in this rendered timeline are artifacts of the profiling resolution, not the schedule itself. In a true 1F1B, each GPU executes its remaining backward passes back-to-back during cool-down.

The timeline shows alternating forward and backward tasks; small rendered gaps are profiling artifacts, not necessarily schedule idle time. section 6.5.4

Reducing the pipeline bubble does not remove every source of idle time. Mixture-of-experts layers also require cross-GPU token exchange, which can leave GPUs waiting for communication. DualPipe uses a folded stage layout and fine-grained scheduling to overlap communication with computation. section 6.5.5 section 6.6

DualPipeV simplifies the folded schedule to one stream of micro-batches: data travels down the first half of the stages and up the second half. An N-stage pipeline uses N/2 GPUs, with two stages per GPU. This saves devices but increases parameter memory per GPU; compared with bidirectional DualPipe, it trades some peak bubble reduction for simpler scheduling. section 6.5.7 section 6.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Traces fill, steady-state, and drain behavior, gives an efficiency expression, and compares interleaved and folded schedules.
Examples
Uses four-GPU timelines with eight micro-batches and an eight-stage folded pipeline on four GPUs.
What you can do after
Enables readers to evaluate pipeline timelines and scheduling trade-offs using bubble size, communication waits, and per-GPU memory.
How it connects
Connects pipeline parallelism with micro-batching, gradient propagation, mixture-of-experts communication, and computation–communication overlap.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.