Pipeline parallelism

As explained in Designing Deep Learning Systems: A guide for software engineers

A distributed training technique that partitions a model into sequential stages executed by different workers, allowing batches to flow through the stages concurrently.

Pipeline parallelism trains a model that cannot fit on one GPU by partitioning its sequential layers into stages on multiple devices. It improves on naive model parallelism: rather than letting stages wait for one batch to traverse the entire network, it keeps stages working concurrently on different pieces of that batch. section 4.1 section 4.4.2

Figure 4.8 (a) An example neural network with sequential layers is partitioned across four accelerators. F_k is the composite forward computation function of the kth cell. Bk is the backpropagation function, which depends on both B_k+1, from the upper layer, and F_k. (b) The naive model parallelism strategy leads to severe under utilization due to the sequential dependency of the network. (c) Pipeline parallelism divides the input minibatch into smaller microbatches, enabling different accelerators to work on different microbatches simultaneously. Gradients are applied synchronously at the end. (Source: figure 2, “GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism,” Huang et al., 2019, arXiv:1811.06965)

The pipeline timeline contrasts sequential stage idleness with microbatches flowing through partitioned layers concurrently; gradients are applied synchronously after the minibatch. section 4.4.2

Start by splitting a neural network into sequential subnetworks and assigning each to a GPU. Divide each input minibatch into smaller microbatches. While an earlier stage processes a later microbatch, downstream stages process earlier ones, overlapping forward and backward work across the available accelerators. This raises device utilization and training throughput compared with sequential model parallelism. section 4.1 section 4.4.2

Pipeline parallelism is appropriate when its capacity benefit outweighs its implementation and operational cost:

• Use it when the whole neural network exceeds the memory of a single device; data parallelism cannot resolve model-size memory limits because every worker needs a complete model copy. section 4.2.1 section 4.4.2
• Prefer data parallelism when the model fits on one GPU and the main problem is training time, because it requires fewer code changes and is easier to understand and debug. section 4.1 section 4.4.3
• Plan the partitioning per model architecture. Converting single-device code requires splitting the network, placing subnetworks on devices, and transferring intermediate outputs; no general splitting method is provided. section 4.1 section 4.4.2

Two-stage transformer deployment

A multi-machine training setup can initialize remote communication, divide transformer encoder layers across two GPUs, and construct a pipeline over those partitions. The training system must provide each worker with its identity and group settings, including worker address, process ID, GPU ID, and group size. section 4.4.2 section 4.4.3

Pipeline code is usually written by data scientists, while platform engineering automates its execution. A training service can allocate workers, establish interworker communication, distribute code and initialized parameters, and pursue high GPU utilization. Because distributed training depends on every worker, monitoring, failover, and recovery are required to limit the cost of a worker failure. section 4.4.3

Treat pipeline parallelism as a specialized response to model capacity, not the default distributed-training option. Begin on one machine; move to distributed training for long-running large-data workloads, choose data parallelism when possible, and choose a pipeline when the model itself cannot load on one GPU. section 4.4.3 section 4.5

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Explains microbatch overlap, model partitioning, PyTorch pipeline setup, and operational support, while deferring any general model-splitting method.

Examples

Uses partitioned sequential networks, a GPU-utilization timeline, and a two-GPU transformer pipeline with remote communication.

What it assumes

Assumes familiarity with neural networks, GPUs, minibatches, and distributed training; introduces pipeline parallelism through model parallelism.

What you can do after

Operate/configure it through a training service that allocates workers, enables communication, distributes parameters, and supports recovery planning.

How it connects

Connects model parallelism, data parallelism, microbatches, distributed training, GPU utilization, and worker failure recovery.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListPipeline parallelism is a distributed-training technique that splits a model into sequential stages and executes microbatches through those stages concurrently.
• 

  liveBookmanning.com
  LLMs in ProductionPipeline parallelism is a distributed-computing technique that partitions a model into sequential stages placed on different devices.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA model-parallel technique that splits a model into stages and overlaps their execution.

Compare how 4 books explain pipeline parallelism →

Related concepts

• model parallelismkind of · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
• GPipeimplemented by · GPipe is a pipeline-parallel training method that partitions a neural network into stages, processes microbatches through them, and accumulates gradients before updating parameters.
• batchinguses · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.
• distributed trainingkind of · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
• tensor parallelismcontrasts with · Tensor parallelism is a model-parallel technique that partitions tensor computations and their associated parameters across multiple devices and combines their partial results.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.