Idle percentage measures how much pipeline execution time devices spend waiting instead of performing useful computation. In pipeline parallelism, sequential processing creates an inactivity bubble: one GPU works while others wait. section 3.3.2
Estimating the bubble
Use Idle Percentage = 1 − m / (m + n − 1), where m is the number of microbatches and n is pipeline depth or GPU count. Four GPUs processing one large batch are idle 75% of the time; four microbatches reduce this to about 43%. section 3.3.2
Use the metric to reason about practical trade-offs:
- Increase microbatches to start downstream devices earlier and improve utilization, but expect cached activation memory to grow linearly. section 3.3.2
- Prefer fewer larger GPUs when possible; memory requirements may force additional devices, increasing idle time. section 3.3.2
Figure 3.6 The bubble problem. When data runs through a broken-up model, the GPUs holding the model weights are underutilized while they wait for their counterparts to process the data. A simple way to reduce this bubble is to use microbatching.
The bubble diagram makes waiting periods visible and shows how microbatching reduces underutilization. section 3.3.2
