Multi-head attention
Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
Natural Language Processing in Action, Second Edition
Multi-head attention computes multiple attention representations in parallel and combines them, potentially using distinct query and source sequences.
- How it is used
- Combines parallel attention representations inside stackable transformers and connects decoder outputs to encoded source sequences during translation.
- How deep it goes
- Works through scoring, head dimensions, concatenation, and projection, then implements weight inspection through PyTorch decoder customization rather than building attention from scratch.
- Examples
- Uses an eight-head transformer configuration, German–English translation samples, customized decoder code, and attention-weight plots.
- What you can do after
- Apply it via PyTorch’s Transformer modules to train a translation model and inspect decoder attention weights.
Read how Natural Language Processing in Action, Second Edition explains multi-head attention →
Build a Text-to-Image Generator (from Scratch)
An attention mechanism that computes attention through multiple separate query, key, and value subspaces to capture different relationships simultaneously.
- How it is used
- Uses multiple attention heads to capture different relationships in text and image inputs.
- How deep it goes
- Works through per-head projections and scaled dot-product calculations, but leaves one text transformer's full MultiHeadedAttention class definition off-page.
- Examples
- Illustrates heads with an ambiguous “bank” pun and image features such as texture and edges.
- What you can do after
- Builds attention heads from scratch in PyTorch and integrates multi-head attention into transformer encoder blocks.
Read how Build a Text-to-Image Generator (from Scratch) explains multi-head attention →
Transformers in Action
Multihead attention is an attention mechanism that computes multiple attention patterns in parallel so a model can represent different relationships among sequence positions.
- How it is used
- Uses parallel attention perspectives to identify multiple relationships among sequence positions in a transformer.
- How deep it goes
- Works from scaled dot-product attention through projected, split heads to a simplified PyTorch implementation.
- Examples
- Illustrates negation in movie sentiment and runs self-attention on batched PyTorch tensors with eight heads.
- What you can do after
- Build it from scratch with a simplified PyTorch multi-head attention module and apply it to self-attention inputs.
Read how Transformers in Action explains multi-head attention →
Learn Generative AI with PyTorch
A self-attention variant that splits representations into multiple attention heads to capture different relationships in parallel.
- How it is used
- Builds context-aware representations within Transformer layers and supports attention in text generation and image denoising.
- How deep it goes
- Works from query, key, and value through per-head shapes and a PyTorch implementation, then shows decoder masking.
- Examples
- Uses a six-token sentence with eight 32-dimensional heads, a wordplay example, and a masked translation decoder.
- What you can do after
- Build it from scratch as a PyTorch attention module and use it in Transformer encoder and decoder layers.
Read how Learn Generative AI with PyTorch explains multi-head attention →
Inside Deep Learning: Math, Algorithms, Models
An attention mechanism that uses multiple attention heads to attend to different contexts or aspects of the input in parallel.
- How it is used
- Uses parallel attention contexts to replace or extend recurrent sequence modeling within transformer architectures.
- How deep it goes
- Explains tensor roles, shapes, head behavior, and PyTorch construction, while deferring detailed transformer additions.
- Examples
- Uses negation detection, a key-value dictionary analogy, tensor shapes, and nn.ModuleList projection code.
- What you can do after
- Apply it via PyTorch by projecting Q, K, and V per head, combining head outputs, and returning fixed-width representations.
Read how Inside Deep Learning: Math, Algorithms, Models explains multi-head attention →
Deep Learning with Python, Third Edition
An attention variant that runs several attention computations in parallel with different parameters to learn different relationships.
- How it is used
- Lets sequence models combine information through multiple learned attention patterns in Transformer encoder and decoder blocks.
- How deep it goes
- Works through per-head projections, scoring, combination, and a reusable Keras layer, then places it in encoder and decoder blocks.
- Examples
- Uses subject-versus-punctuation attention, a context-dependent station example, and Keras encoder and decoder implementations.
- What you can do after
- Apply it via Keras MultiHeadAttention in encoder self-attention and masked decoder self-attention or cross-attention.
Read how Deep Learning with Python, Third Edition explains multi-head attention →
Graph Neural Networks in Action
An attention technique that computes multiple attention operations in parallel and combines their outputs.
- How it is used
- Uses multiple attention heads in GAT layers to learn which neighboring nodes matter when updating node features.
- How deep it goes
- Explains neighbor attention scores and aggregation, then shows head-dependent layer dimensions without a rigorous derivation.
- Examples
- Shows a GATConv architecture with two heads in its first layer and one head in its output layer.
- What you can do after
- Apply it via PyTorch Geometric GATConv by setting heads and matching subsequent layers to the resulting feature dimensions.
Read how Graph Neural Networks in Action explains multi-head attention →
Build a Large Language Model (From Scratch)
An attention technique that computes several independent attention heads in parallel and combines their outputs to capture different relationships in the same sequence.
- How it is used
- Deploys parallel causal attention heads inside transformer blocks to analyze relationships among sequence elements.
- How deep it goes
- Decomposes heads, learned projections, concatenation, tensor shapes, and wrapper code, then notes batched multiplication without implementing it.
- Examples
- Demonstrates two heads producing four features per token from a two-sequence, six-token batch and supplies a dimensionality exercise.
- What you can do after
- Builds multi-head attention with PyTorch modules, verifies output shapes, and places it inside a GPT transformer block.
Read how Build a Large Language Model (From Scratch) explains multi-head attention →
Deep Learning with R, Third Edition
MultiHeadAttention is a Keras layer that computes attention through multiple parallel attention heads and combines their outputs.
- How it is used
- Uses parallel attention heads to preserve distinct source-token relationships in sequence models and improve handling of long-range dependencies.
- How deep it goes
- Explains projections, scaled scores, softmax weighting, concatenation, and encoder integration, without deriving the underlying mathematics.
- Examples
- Uses R/Keras listings, source-target sequence attention, and the contextual example of “station” attending to “train.”
- What you can do after
- Build it from scratch or operate a Keras layer for cross-sequence attention, self-attention, and Transformer encoder blocks.
Read how Deep Learning with R, Third Edition explains multi-head attention →
Time Series Forecasting Using Foundation Models
An attention mechanism with multiple parallel attention heads that combines information from different representation subspaces.
- How it is used
- Uses parallel attention heads to learn diverse temporal relationships inside transformer-based forecasting models.
- How deep it goes
- Explains roles, masking, and architecture connections, but defers the mechanism's mathematical details.
- Examples
- Illustrates trend and seasonal heads, nearer historical tokens, masked decoder inputs, and TimesFM input patches.
- What you can do after
- Evaluate/understand how attention heads, causal masks, and encoder representations support transformer forecasts.
Read how Time Series Forecasting Using Foundation Models explains multi-head attention →
Machine Learning Algorithms in Depth
Multihead attention runs multiple attention mechanisms in parallel and combines their output representations.
- How it is used
- Builds transformer encoder–decoder and classification architectures from parallel attention representations.
- How deep it goes
- Defines parallel heads, concatenation, and dense transformation, but does not derive attention equations or head computations.
- Examples
- Illustrates translation attention, transformer architecture, multihead-attention structure, and Keras text classification.
- What you can do after
- Apply multihead attention through Keras in a transformer text classifier.
Read how Machine Learning Algorithms in Depth explains multi-head attention →
Deep Learning with PyTorch, Second Edition
An attention mechanism that runs several attention heads in parallel, each learning different projections and relationships.
- How it is used
- Uses parallel attention heads inside transformer decoder and encoder-decoder architectures to capture diverse sequence relationships.
- How deep it goes
- Explains learned projections, head specialization, concatenation, and surrounding components, while stopping short of detailed attention-score mathematics.
- Examples
- Demonstrates a PyTorch TransformerBlock and a stacked character-level Transformer that generates sample names.
- What you can do after
- Builds it from scratch in PyTorch as part of a transformer block and connects it to sequence generation.
Read how Deep Learning with PyTorch, Second Edition explains multi-head attention →
Optimization Algorithms: AI techniques for design, planning, and control problems
Multi-head self-attention computes several attention operations in parallel so a sequence representation can capture different relationships among its elements.
- How it is used
- Uses parallel attention focuses to capture different relationships in graph neighborhoods and Transformer models.
- How deep it goes
- Explains multiple heads, neighborhood attention, softmax normalization, averaging, and final nonlinearities without implementation details.
- Examples
- Illustrates three attention heads aggregating features for node 5 before averaging their embeddings.
- What you can do after
- Evaluate how multiple attention heads combine distinct neighborhood perspectives into a prediction-ready representation.
The AI Pocket Book
A transformer attention mechanism that splits representations into multiple heads to attend to different relationships in parallel.
- How it is used
- Uses parallel attention heads to improve contextualized representations during transformer-based language modeling.
- How deep it goes
- Defines separate vector chunks and explains their specialization, but stops short of detailing attention calculations or head outputs.
- Examples
- Illustrates specialization with hypothetical animal-related and tree-related embedding segments.
- What you can do after
- Evaluate how separate attention heads can specialize representations and outperform a single whole-vector attention head.
Related concepts
- graph attention networkused by · A graph attention network is a graph neural network that uses attention weights to aggregate information from neighboring nodes, allowing their contributions to differ.
- grouped-query attentionequivalent to · Grouped-query attention partitions query heads into groups whose members share key and value projections, reducing attention computation and memory use.
- attentionkind of · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
- attention headhas part · An attention head is an independently parameterized attention computation that evaluates relationships among input elements and combines information from them into a contextualized representation.
- transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
- encoderpart of · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.