Sequence parallelism

As explained in LLMs in Production

Sequence parallelism is a distributed-computing technique that partitions neural-network activations along the sequence dimension to reduce redundant storage.

Sequence parallelism partitions neural-network activations along the sequence dimension, reducing redundant storage. It complements tensor parallelism by targeting memory-limiting normalization and dropout blocks that tensor parallelism does not efficiently distribute. section 3.3.2

Use the technique when activation memory, rather than computation, limits transformer efficiency:

• Combine it with tensor parallelism: tensor parallelism distributes computationally heavy MLP and attention layers, while sequence parallelism distributes memory-limiting work. section 3.3.2
• Expect experimentation: hardware and dataset dimensions determine the best partitioning strategy. section 3.3.2

Figure 3.5 Combining tensor parallelism that focuses on computational heavy layers with sequence parallelism to reduce memory overhead to create a fully parallel process for the entire transformer

Use the diagram to see how tensor parallelism handles computation-heavy layers while sequence parallelism reduces activation-memory overhead across the transformer. section 3.3.2

How LLMs in Production covers it

How deep it goes

Defines activation partitioning and its complementarity with tensor parallelism, but omits implementation steps.

Examples

Illustrates combined tensor and sequence parallelism through a transformer-parallelization diagram.

What you can do after

Apply it alongside tensor parallelism to reduce transformer activation memory with minimal additional computational overhead.

How it connects

Connects tensor parallelism, transformer models, normalization, dropout, activation recomputation, and pipeline parallelism.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

• tensor parallelismuses · Tensor parallelism is a model-parallel technique that partitions tensor computations and their associated parameters across multiple devices and combines their partial results.
• model parallelismkind of · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.