Tensor parallelism
Tensor parallelism is a model-parallel technique that partitions tensor computations and their associated parameters across multiple devices and combines their partial results.
LLMs in Production
Tensor parallelism is a distributed-computing technique that partitions tensor computations across multiple processors and combines their results.
- How it is used
- Uses distributed tensor computation to fit large LLM workloads across processors and reduce calculation time.
- How deep it goes
- Explains row and column partitioning, recombination, dimension selection, and data-movement costs without detailing a complete implementation.
- Examples
- Uses a large matrix split by rows or columns and a visual comparison of equivalent tensor-parallel decompositions.
- What you can do after
- Apply tensor parallelism by partitioning activations across processors and combining their partial results.
Sutskever's List
Tensor parallelism is a distributed-training technique that partitions tensor computations and their associated parameters across devices.
- How it is used
- Distributes tensor computations and associated model data across devices to support large-scale training.
- How deep it goes
- Defines sharding and gives tensor-axis examples, without detailing the mechanics of tensor slicing.
- Examples
- Uses AlexNet’s hand-built sharding, Mesh-TensorFlow’s axis choices, and Megatron-LM transformer training as examples.
- What you can do after
- Understand how tensor-axis partitioning differs from batch splitting and pipeline-stage partitioning.
Deep Learning with PyTorch, Second Edition
A technique that partitions layer parameters or computation across devices.
- How it is used
- Scales wide models and oversized layers by distributing their parameters and computation across devices.
- How deep it goes
- Explains parameter partitioning, intermediate communication, and row-wise or column-wise splits, but defers detailed strategy reasoning.
- Examples
- Uses split linear-layer weights, two-device computation, and a pipeline-versus-tensor granularity figure.
- What you can do after
- Evaluate whether tensor parallelism fits a model’s width, layer size, memory limits, and device communication pattern.
Read how Deep Learning with PyTorch, Second Edition explains tensor parallelism →
Related concepts
- pipeline parallelismcontrasts with · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
- matrix-matrix multiplicationuses · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.
- dataset splitequivalent to · A dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or testing.
- vLLMused by · vLLM is an open-source library and serving engine for high-throughput, memory-efficient inference of large language models.
- allreduceuses · Allreduce is a collective communication operation that combines corresponding values across participating processes or devices and distributes the resulting aggregate to every participant.
- model parallelismkind of · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.