N-dimensional parallelism

As explained in LLMs in Production

3D parallelism is a distributed-training strategy that combines data parallelism, tensor parallelism, and pipeline parallelism.

3D parallelism combines data parallelism (DP), tensor parallelism (TP), and pipeline parallelism (PP) to distribute LLM training across GPUs. DP increases throughput, TP increases speed, and PP makes training possible when the model does not fit on one device. section 3.3.2 section 3.5

A practical configuration follows the communication and memory trade-offs of each dimension:

• Start with at least eight GPUs, because each of the three dimensions requires at least two GPUs. section 3.3.2
• Place tensor-parallel GPUs close together, preferably within the same node, because TP creates the largest communication overhead. section 3.3.2
• Spread pipeline stages across nodes when necessary, since PP has the lowest communication volume of the three methods. section 3.3.2
• Use microbatching to reduce pipeline bubbles, while accounting for the resulting linear growth in cached activation memory. section 3.3.2

Why the combination helps

TP partitions the model to fit device memory, allowing PP to operate with smaller effective batch sizes. Together, they improve communication between DP nodes and can produce better utilization than any single method alone. section 3.3.2

Figure 3.6 The bubble problem. When data runs through a broken-up model, the GPUs holding the model weights are underutilized while they wait for their counterparts to process the data. A simple way to reduce this bubble is to use microbatching.

Use the pipeline-bubble diagram to see why GPUs can remain underutilized while stages wait, and how microbatching reduces that idle time. section 3.3.2

How LLMs in Production covers it

How deep it goes

Explains GPU requirements, placement, communication, memory effects, and pipeline bubbles, but does not provide a complete deployment configuration.

Examples

Uses an eight-GPU starting configuration, tensor and pipeline placement choices, microbatching, PipeDream, and a pipeline-bubble diagram.

What it assumes

Assumes familiarity with data, tensor, and pipeline parallelism as separate distributed-training practices.

What you can do after

Configure distributed LLM training by assigning GPUs across DP, TP, and PP while balancing communication, memory, and utilization.

How it connects

Connects data parallelism, tensor parallelism, pipeline parallelism, microbatching, model compression, and distributed training.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA distributed-training approach combining multiple parallelism strategies along different dimensions.

Compare how 2 books explain n-dimensional parallelism →

Related concepts

• data paralleluses · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
• tensor parallelismhas part · Tensor parallelism is a model-parallel technique that partitions tensor computations and their associated parameters across multiple devices and combines their partial results.
• model parallelismuses · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
• distributed trainingkind of · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.