Pipeline parallelism

As explained in Deep Learning with PyTorch, Second Edition

A model-parallel technique that splits a model into stages and overlaps their execution.

Pipeline parallelism splits a model into sequential stages placed on different devices, then divides each input batch into microbatches. Different stages process different microbatches at the same time, reducing the idle time found in basic model parallelism. section 16.5.1

Figure 16.7 Pipeline parallelism with three microbatches involves processing each microbatch and then communicating it to the next device, as indicated by the arrows. This approach enables concurrent processing across devices.

The three-device diagram shows microbatches moving between stages while computation overlaps across devices. section 16.5.1

A practical pipeline setup involves these decisions:

• Split the model at suitable layer boundaries and assign each stage to a device. section 16.5.1
• Choose microbatches so stages can work concurrently rather than waiting for the entire batch. section 16.5.1
• Balance stage workloads carefully; uneven computation creates bottlenecks and stragglers. section 16.5.1
• Use torch.distributed.pipelining utilities to define split points, build stages, and execute a schedule. section 16.5.1
• Prefer it for skinnier, deeper models; wider, shallower models or layers too large for one device favor tensor parallelism. section 16.5.3

Defining and running a pipeline

A PyTorch model can mark split points with a split_spec attribute, such as placing SplitPoint.BEGINNING before successive layers. The pipeline is then built from a sample microbatch, a stage is assigned by rank, and the schedule is stepped differently on the first, middle, and final ranks. section 16.5.1

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes

Explains stages, microbatches, tradeoffs, architecture fit, and PyTorch setup, but defers detailed schedule mechanics and performance tuning.

Examples

Uses a three-device microbatch diagram and a PyTorch MLP with split points, stages, and rank-specific schedule steps.

What it assumes

Assumes basic model and multi-GPU parallelism concepts while introducing stages, microbatches, split points, and rank-specific execution.

What you can do after

Build it from scratch with torch.distributed.pipelining by marking split points, constructing stages, and running a pipeline schedule.

How it connects

Connects model parallelism, tensor parallelism, data parallelism, distributed training, and n-dimensional parallelism.

liveBookmanning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListPipeline parallelism is a distributed-training technique that splits a model into sequential stages and executes microbatches through those stages concurrently.
• 

  liveBookmanning.com
  Designing Deep Learning Systems: A guide for software engineersA distributed training technique that partitions a model into sequential stages executed by different workers, allowing batches to flow through the stages concurrently.
• 

  liveBookmanning.com
  LLMs in ProductionPipeline parallelism is a distributed-computing technique that partitions a model into sequential stages placed on different devices.

Compare how 4 books explain pipeline parallelism →

Related concepts

• model parallelismkind of · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
• GPipeimplemented by · GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.
• batchinguses · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.
• distributed trainingkind of · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
• tensor parallelismcontrasts with · Tensor parallelism is a model-parallel technique that partitions tensor computations and their associated parameters across multiple devices and combines their partial results.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.