How deeply does LLMs in Production cover pipeline parallelism?

Explains vertical partitioning, device transfers, idle bubbles, and microbatching, but does not detail scheduling algorithms.

Pipeline parallelism

As explained in LLMs in Production

Pipeline parallelism is a distributed-computing technique that partitions a model into sequential stages placed on different devices.

When a model is too large for one GPU, pipeline parallelism splits it vertically into sequential stages on different GPUs. Input data moves from one device to the next until the full model has run, making execution possible rather than merely faster. section 3.3.2 section 3.5

The practical trade-offs are straightforward:

A naive one-batch pipeline creates a large utilization bubble: while one GPU processes its stage, other GPUs wait. section 3.3.2
Microbatching reduces the bubble by letting the first GPU forward smaller batches sooner and continue working while later stages process earlier batches. section 3.3.2
Pipeline parallelism can combine with tensor and data parallelism; together, data, tensor, and pipeline parallelism form 3D parallelism. section 3.5

How LLMs in Production covers it

liveBook manning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Sutskever's ListPipeline parallelism is a distributed-training technique that splits a model into sequential stages and executes microbatches through those stages concurrently.
liveBook manning.com

Designing Deep Learning Systems: A guide for software engineersA distributed training technique that partitions a model into sequential stages executed by different workers, allowing batches to flow through the stages concurrently.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA model-parallel technique that splits a model into stages and overlaps their execution.

Compare how 4 books explain pipeline parallelism →

Related concepts

model parallelismkind of · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
GPipeimplemented by · GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.
batchinguses · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.
distributed trainingkind of · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
tensor parallelismcontrasts with · Tensor parallelism is a model-parallel technique that partitions tensor computations and their associated parameters across multiple devices and combines their partial results.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.