Tensor parallelism

As explained in Deep Learning with PyTorch, Second Edition

A technique that partitions layer parameters or computation across devices.

Tensor parallelism is a form of model parallelism that splits a layer’s parameters or tensors across multiple devices. Unlike pipeline parallelism, which divides a model between modules, it divides computation inside the modules themselves. section 16.5.2

Each device works on its segment of a layer, such as part of a linear layer’s weights and biases. Devices then exchange intermediate results so the combined computation produces the correct output shape. section 16.5.2

Tensor parallelism is especially useful when:

• A model is wide and shallow, with relatively few layers but many parameters in each layer. section 16.5.3
• A single layer is too large to fit in one device’s memory. section 16.5.3
• Fast communication is available between devices, since splitting tensors requires exchanging intermediate results. section 16.5.2 section 16.6

Splitting a linear layer

A linear layer can divide its weights row-wise or column-wise across two devices. Each device performs matrix multiplication with the input using its weight segment, then communication combines the partial results into the layer’s expected output. section 16.5.2

Figure 16.8 Pipeline parallelism (top) compared to tensor parallelism (bottom). Pipeline parallelism splits at the module granularity, while tensor parallelism splits at the module parameter (tensor) granularity.

Compare pipeline parallelism’s module-level split with tensor parallelism’s parameter-level split. section 16.5.2

Tensor parallelism can also combine with data, pipeline, and other parallelism strategies. DeviceMesh helps define the device groups and communication patterns needed for these multidimensional arrangements. section 16.6

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes

Explains parameter partitioning, intermediate communication, and row-wise or column-wise splits, but defers detailed strategy reasoning.

Examples

Uses split linear-layer weights, two-device computation, and a pipeline-versus-tensor granularity figure.

What it assumes

Builds from model and data parallelism before distinguishing tensor-level partitioning from module-level partitioning.

What you can do after

Evaluate whether tensor parallelism fits a model’s width, layer size, memory limits, and device communication pattern.

How it connects

Connects model parallelism, pipeline parallelism, data parallelism, n-dimensional parallelism, and DeviceMesh.

liveBookmanning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionTensor parallelism is a distributed-computing technique that partitions tensor computations across multiple processors and combines their results.
• 

  liveBookmanning.com
  Sutskever's ListTensor parallelism is a distributed-training technique that partitions tensor computations and their associated parameters across devices.

Compare how 3 books explain tensor parallelism →

Related concepts

• pipeline parallelismcontrasts with · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
• matrix-matrix multiplicationuses · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.
• dataset splitequivalent to · A dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or testing.
• vLLMused by · vLLM is an open-source library and serving engine for high-throughput, memory-efficient inference of large language models.
• allreduceuses · Allreduce is a collective communication operation that combines corresponding values from all participating processes or devices and distributes the resulting aggregate to every participant.
• model parallelismkind of · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.