N-dimensional parallelism
N-dimensional parallelism is a distributed-training strategy that combines multiple parallelism methods across distinct dimensions to improve computational or memory efficiency.
LLMs in Production
3D parallelism is a distributed-training strategy that combines data parallelism, tensor parallelism, and pipeline parallelism.
- How it is used
- Coordinates data, tensor, and pipeline parallelism to make large-scale LLM training more feasible and efficient across GPUs.
- How deep it goes
- Explains GPU requirements, placement, communication, memory effects, and pipeline bubbles, but does not provide a complete deployment configuration.
- Examples
- Uses an eight-GPU starting configuration, tensor and pipeline placement choices, microbatching, PipeDream, and a pipeline-bubble diagram.
- What you can do after
- Configure distributed LLM training by assigning GPUs across DP, TP, and PP while balancing communication, memory, and utilization.
Read how LLMs in Production explains n-dimensional parallelism →
Deep Learning with PyTorch, Second Edition
A distributed-training approach combining multiple parallelism strategies along different dimensions.
- How it is used
- Combines parallelism dimensions to scale training across hundreds or thousands of GPUs while tuning communication and efficiency.
- How deep it goes
- Explains strategy selection and DeviceMesh process groups, but stops short of a complete production implementation.
- Examples
- Uses a 2 × 4 DeviceMesh, replica parameter synchronization, model-partition computation, LLaMA 3, and TorchTitan.
- What you can do after
- Evaluate and combine parallelism strategies for model size, hardware placement, communication cost, and training throughput.
Read how Deep Learning with PyTorch, Second Edition explains n-dimensional parallelism →
Related concepts
- data paralleluses · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
- tensor parallelismhas part · Tensor parallelism is a model-parallel technique that partitions tensor computations and their associated parameters across multiple devices and combines their partial results.
- model parallelismuses · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
- distributed trainingkind of · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.