GPipe

As explained in Designing Deep Learning Systems: A guide for software engineers

A pipeline-parallel training implementation that splits batches into microbatches and accumulates their gradients before updating model parameters.

GPipe improves model-parallel training by splitting each batch into microbatches and overlapping their computation across subnetworks on different GPUs. It accumulates gradients for all microbatches before synchronously updating parameters, keeping accelerators busy instead of leaving most idle. section 4.4.2

A GPipe training workflow:

• Partition the network across GPUs. section 4.4.2
• Divide each training batch into smaller microbatches and pipeline their forward passes. section 4.4.2
• Run backward passes using the same parameters, accumulate every microbatch’s gradients, then apply one synchronized update. section 4.4.2

Four-stage pipeline

A four-subnetwork model places one subnetwork on each GPU. Four equal microbatches move through the stages, allowing different GPUs to compute simultaneously while gradients accumulate until the training step ends. section 4.4.2

Figure 4.8 (a) An example neural network with sequential layers is partitioned across four accelerators. F_k is the composite forward computation function of the kth cell. Bk is the backpropagation function, which depends on both B_k+1, from the upper layer, and F_k. (b) The naive model parallelism strategy leads to severe under utilization due to the sequential dependency of the network. (c) Pipeline parallelism divides the input minibatch into smaller microbatches, enabling different accelerators to work on different microbatches simultaneously. Gradients are applied synchronously at the end. (Source: figure 2, “GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism,” Huang et al., 2019, arXiv:1811.06965)

Use the pipeline diagram to compare idle-heavy naive model parallelism with microbatch overlap and synchronous gradient application. section 4.4.2

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Explains partitioning, forward/backward scheduling, gradient accumulation, and synchronized updates; stops at a high-level workflow.

Examples

Uses a four-subnetwork, four-GPU schedule and a transformer model split across two GPUs.

What it assumes

Assumes familiarity with model partitioning, GPUs, forward passes, and backward passes.

What you can do after

Build it from scratch using PyTorch model partitioning and remote-communication patterns.

How it connects

Connects model parallelism, pipeline parallelism, microbatches, forward passes, backward passes, and gradient updates.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListGPipe is a system for training neural networks by partitioning computation into pipeline stages across devices.

Compare how 2 books explain GPipe →

Related concepts

• pipeline parallelismimplements · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
• AmoebaNetuses · AmoebaNet is a convolutional neural network architecture discovered through neural architecture search for image recognition.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.