How deeply does LLMs in Production cover tensor parallelism?

Explains row and column partitioning, recombination, dimension selection, and data-movement costs without detailing a complete implementation.

Tensor parallelism

As explained in LLMs in Production

Tensor parallelism is a distributed-computing technique that partitions tensor computations across multiple processors and combines their results.

Tensor parallelism splits tensor computations across processors, runs the pieces concurrently, and combines their results. It helps when a matrix or tensor cannot fit on one processor or its calculations become a throughput bottleneck. section 3.3.2

A practical tensor-parallel workflow follows these decisions:

Choose a tensor dimension to partition, commonly by rows or columns. section 3.3.2
Prefer a dimension with more elements than the number of processors so the available resources are fully used. section 3.3.2
Run the partial calculations concurrently, then communicate and combine the results. section 3.3.2
Treat the partition choice as a trade-off: different dimensions impose different data-movement costs. section 3.3.2

Splitting a large matrix

Suppose Y is too large for one processor. Split Y by columns or rows, perform the corresponding matrix calculations on separate processors, and combine the partial outputs to obtain the same result. The same principle extends from matrices to tensors with more than two dimensions. section 3.3.2

How LLMs in Production covers it

liveBook manning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Sutskever's ListTensor parallelism is a distributed-training technique that partitions tensor computations and their associated parameters across devices.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA technique that partitions layer parameters or computation across devices.

Compare how 3 books explain tensor parallelism →

Related concepts

pipeline parallelismcontrasts with · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
matrix-matrix multiplicationuses · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.
dataset splitequivalent to · A dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or testing.
vLLMused by · vLLM is an open-source library and serving engine for high-throughput, memory-efficient inference of large language models.
allreduceuses · Allreduce is a collective communication operation that combines corresponding values across participating processes or devices and distributes the resulting aggregate to every participant.
model parallelismkind of · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.