liveBook · Manning

Tensor parallelism

Tensor parallelism is a model-parallel technique that partitions tensor computations or associated parameters across multiple devices and combines their partial results.

How 4 Manning books explain it

LLMs in Production

Tensor parallelism is a distributed-computing technique that partitions tensor computations across multiple processors and combines their results.

How it is used
Uses distributed tensor computation to fit large LLM workloads across processors and reduce calculation time.
How deep it goes
Explains row and column partitioning, recombination, dimension selection, and data-movement costs without detailing a complete implementation.
Examples
Uses a large matrix split by rows or columns and a visual comparison of equivalent tensor-parallel decompositions.
What you can do after
Apply tensor parallelism by partitioning activations across processors and combining their partial results.

Read how LLMs in Production explains tensor parallelism →

CUDA for LLMs

Tensor parallelism distributes parts of a tensor computation across multiple devices and combines their results.

How it is used
Uses tensor parallelism to fit wide model layers across GPUs and examine communication-limited scaling.
How deep it goes
Works through sharding, per-GPU allocation, NCCL reduction, and measured scaling, while noting production overlap of communication and computation.
Examples
Uses an eight-GPU FP16 matrix multiplication and H100 SXM timing sweep to illustrate sharding and collective overhead.
What you can do after
Evaluate whether a wide layer benefits from tensor parallelism by comparing memory capacity, local computation, and end-to-end collective cost.

Read how CUDA for LLMs explains tensor parallelism →

Sutskever's List

Tensor parallelism is a distributed-training technique that partitions tensor computations and their associated parameters across devices.

How it is used
Distributes tensor computations and associated model data across devices to support large-scale training.
How deep it goes
Defines sharding and gives tensor-axis examples, without detailing the mechanics of tensor slicing.
Examples
Uses AlexNet’s hand-built sharding, Mesh-TensorFlow’s axis choices, and Megatron-LM transformer training as examples.
What you can do after
Understand how tensor-axis partitioning differs from batch splitting and pipeline-stage partitioning.

Read how Sutskever's List explains tensor parallelism →

Deep Learning with PyTorch, Second Edition

A technique that partitions layer parameters or computation across devices.

How it is used
Scales wide models and oversized layers by distributing their parameters and computation across devices.
How deep it goes
Explains parameter partitioning, intermediate communication, and row-wise or column-wise splits, but defers detailed strategy reasoning.
Examples
Uses split linear-layer weights, two-device computation, and a pipeline-versus-tensor granularity figure.
What you can do after
Evaluate whether tensor parallelism fits a model’s width, layer size, memory limits, and device communication pattern.

Read how Deep Learning with PyTorch, Second Edition explains tensor parallelism →

Also covered in

What to know first

Related concepts

Questions about tensor parallelism

What is tensor parallelism?

Tensor parallelism is a model-parallel technique that partitions tensor computations or associated parameters across multiple devices and combines their partial results.

Which Manning books explain tensor parallelism?

LLMs in Production, CUDA for LLMs, Sutskever's List and Deep Learning with PyTorch, Second Edition.

What should I know before learning tensor parallelism?

Manning books that teach tensor parallelism usually introduce language model, neural network layer, matrix-matrix multiplication and throughput first.

What is the difference between tensor parallelism and pipeline parallelism?

Tensor parallelism is a model-parallel technique that partitions tensor computations or associated parameters across multiple devices and combines their partial results. Pipeline parallelism is a form of model parallelism that divides computation into sequential stages assigned to different devices, allowing multiple inputs or microbatches to progress through the stages concurrently.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.