Breadth-first pipeline parallelism reorders micro-batches to process them in parallel across the earliest model stages, layer by layer, rather than following the classic one-forward, one-backward schedule. The arrangement overlaps computation with communication and works naturally with fully sharded data parallelism. With small per-GPU batches, it achieved up to 43% higher throughput than Megatron’s schedule on a 52-billion-parameter model. section 6.3.3
Breadth-first pipeline parallelism
Breadth-first pipeline parallelism is a pipeline scheduling strategy that processes micro-batches across early model stages in a breadth-first order to overlap computation and communication.
How Sutskever's List covers it
- How deep it goes
- Describes micro-batch ordering, contrasts it with 1F1B scheduling, and reports a throughput comparison.
- Examples
- Compares throughput with Megatron’s schedule on a 52-billion-parameter model using small per-GPU batches.
- What you can do after
- Understand how reordering micro-batches can increase overlap and pipeline throughput.
- How it connects
- Connects pipeline parallelism with 1F1B scheduling, fully sharded data parallelism, and Megatron.
Related concepts
- pipeline parallelismkind of · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.