Tensor parallelism
Tensor parallelism is a model-parallel technique that partitions tensor computations or associated parameters across multiple devices and combines their partial results.
LLMs in Production
Tensor parallelism is a distributed-computing technique that partitions tensor computations across multiple processors and combines their results.
- How it is used
- Uses distributed tensor computation to fit large LLM workloads across processors and reduce calculation time.
- How deep it goes
- Explains row and column partitioning, recombination, dimension selection, and data-movement costs without detailing a complete implementation.
- Examples
- Uses a large matrix split by rows or columns and a visual comparison of equivalent tensor-parallel decompositions.
- What you can do after
- Apply tensor parallelism by partitioning activations across processors and combining their partial results.
CUDA for LLMs
Tensor parallelism distributes parts of a tensor computation across multiple devices and combines their results.
- How it is used
- Uses tensor parallelism to fit wide model layers across GPUs and examine communication-limited scaling.
- How deep it goes
- Works through sharding, per-GPU allocation, NCCL reduction, and measured scaling, while noting production overlap of communication and computation.
- Examples
- Uses an eight-GPU FP16 matrix multiplication and H100 SXM timing sweep to illustrate sharding and collective overhead.
- What you can do after
- Evaluate whether a wide layer benefits from tensor parallelism by comparing memory capacity, local computation, and end-to-end collective cost.
Sutskever's List
Tensor parallelism is a distributed-training technique that partitions tensor computations and their associated parameters across devices.
- How it is used
- Distributes tensor computations and associated model data across devices to support large-scale training.
- How deep it goes
- Defines sharding and gives tensor-axis examples, without detailing the mechanics of tensor slicing.
- Examples
- Uses AlexNet’s hand-built sharding, Mesh-TensorFlow’s axis choices, and Megatron-LM transformer training as examples.
- What you can do after
- Understand how tensor-axis partitioning differs from batch splitting and pipeline-stage partitioning.
Deep Learning with PyTorch, Second Edition
A technique that partitions layer parameters or computation across devices.
- How it is used
- Scales wide models and oversized layers by distributing their parameters and computation across devices.
- How deep it goes
- Explains parameter partitioning, intermediate communication, and row-wise or column-wise splits, but defers detailed strategy reasoning.
- Examples
- Uses split linear-layer weights, two-device computation, and a pipeline-versus-tensor granularity figure.
- What you can do after
- Evaluate whether tensor parallelism fits a model’s width, layer size, memory limits, and device communication pattern.
Read how Deep Learning with PyTorch, Second Edition explains tensor parallelism →
Also covered in
- Deep Learning with JAX
- Self-Hosting LLMs at ScaleIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- neural network layerA neural network layer is a computational component or stage that transforms inputs into outputs within a neural network.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- throughputThroughput is the rate at which a system processes or completes work, data, or requests over a specified period.
Related concepts
- pipeline parallelismcontrasts with · Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently.
- weight matrixuses · A weight matrix is a matrix of learned numerical parameters that connects or transforms representations in a model.
- matrix-matrix multiplicationuses · Matrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- communicationuses · Communication is the exchange of information, meaning, or signals between people, groups, agents, devices, or software components.
- GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and other data-parallel workloads.
- allreduceuses · Allreduce is a collective communication operation that combines corresponding values from participating processes or devices and distributes the result to every participant.
Questions about tensor parallelism
What is tensor parallelism?
Tensor parallelism is a model-parallel technique that partitions tensor computations or associated parameters across multiple devices and combines their partial results.
Which Manning books explain tensor parallelism?
LLMs in Production, CUDA for LLMs, Sutskever's List and Deep Learning with PyTorch, Second Edition.
What should I know before learning tensor parallelism?
Manning books that teach tensor parallelism usually introduce language model, neural network layer, matrix-matrix multiplication and throughput first.
What is the difference between tensor parallelism and pipeline parallelism?
Tensor parallelism is a model-parallel technique that partitions tensor computations or associated parameters across multiple devices and combines their partial results. Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



