liveBook · Manning

Multi-head attention

Multi-head attention is an attention mechanism that performs multiple attention computations in parallel using distinct learned projections and combines their outputs to represent different relationships.

How 15 Manning books explain it

Natural Language Processing in Action, Second Edition

Multi-head attention computes multiple attention representations in parallel and combines them, potentially using distinct query and source sequences.

How it is used
Uses parallel attention heads to form transformer representations for language and translation.
How deep it goes
Explains scaled attention, head concatenation, output projection, and decoder customization rather than implementing attention from scratch.
Examples
Uses an eight-head transformer configuration and German–English translation with visualized decoder attention weights.
What you can do after
Apply it via PyTorch by customizing a transformer decoder to retain and inspect attention weights.

Read how Natural Language Processing in Action, Second Edition explains multi-head attention →

Build a Text-to-Image Generator (from Scratch)

An attention mechanism that computes attention through multiple separate query, key, and value subspaces to capture different relationships simultaneously.

How it is used
Uses multiple attention heads to capture different relationships in text and image inputs.
How deep it goes
Works through per-head projections and scaled dot-product calculations, but leaves one text transformer's full MultiHeadedAttention class definition off-page.
Examples
Illustrates heads with an ambiguous “bank” pun and image features such as texture and edges.
What you can do after
Builds attention heads from scratch in PyTorch and integrates multi-head attention into transformer encoder blocks.

Read how Build a Text-to-Image Generator (from Scratch) explains multi-head attention →

Transformers in Action

Multihead attention is an attention mechanism that computes multiple attention patterns in parallel so a model can represent different relationships among sequence positions.

How it is used
Uses parallel attention patterns to capture multiple relationships and long-range dependencies in transformer sequences.
How deep it goes
Explains attention calculations and encoder placement, then illustrates projections, head dimensions, and a layer invocation.
Examples
Illustrates negation in sentiment classification and a 512-dimensional self-attention call with eight heads.
What you can do after
Apply it via a PyTorch-style multi-head layer to a batch of sequences, using the same input for queries, keys, and values.

Read how Transformers in Action explains multi-head attention →

Learn Generative AI with PyTorch

A self-attention variant that splits representations into multiple attention heads to capture different relationships in parallel.

How it is used
Builds context-aware representations within Transformer layers and supports attention in text generation and image denoising.
How deep it goes
Works from query, key, and value through per-head shapes and a PyTorch implementation, then shows decoder masking.
Examples
Uses a six-token sentence with eight 32-dimensional heads, a wordplay example, and a masked translation decoder.
What you can do after
Build it from scratch as a PyTorch attention module and use it in Transformer encoder and decoder layers.

Read how Learn Generative AI with PyTorch explains multi-head attention →

Inside Deep Learning: Math, Algorithms, Models

An attention mechanism that uses multiple attention heads to attend to different contexts or aspects of the input in parallel.

How it is used
Extends attention inside the vanilla transformer to detect multiple input patterns in parallel.
How deep it goes
Defines query, key, value, and output shapes, then sketches per-head projections and concatenation without detailing the full attention calculation.
Examples
Illustrates pattern detection with “not” preceding “good” and compares attention queries with Python dictionary lookups.
What you can do after
Build a multi-head layer by projecting queries, keys, and values per head, concatenating responses, and mapping them back to D dimensions.

Read how Inside Deep Learning: Math, Algorithms, Models explains multi-head attention →

Deep Learning with Python, Third Edition

An attention variant that runs several attention computations in parallel with different parameters to learn different relationships.

How it is used
Uses multiple attention heads to enrich sequence representations in Transformer encoders and decoders.
How deep it goes
Works through projections, attention scores, head concatenation, and Keras encoder and decoder applications.
Examples
Illustrates heads attending to sentence subjects and punctuation, and self-attention relating “station” to “train.”
What you can do after
Apply it via Keras MultiHeadAttention for encoder self-attention and masked decoder attention.

Read how Deep Learning with Python, Third Edition explains multi-head attention →

Graph Neural Networks in Action

An attention technique that computes multiple attention operations in parallel and combines their outputs.

How it is used
Uses multiple attention heads in GAT layers to learn which neighboring nodes matter when updating node features.
How deep it goes
Explains neighbor attention scores and aggregation, then shows head-dependent layer dimensions without a rigorous derivation.
Examples
Shows a GATConv architecture with two heads in its first layer and one head in its output layer.
What you can do after
Apply it via PyTorch Geometric GATConv by setting heads and matching subsequent layers to the resulting feature dimensions.

Read how Graph Neural Networks in Action explains multi-head attention →

Build a Large Language Model (From Scratch)

An attention technique that computes several independent attention heads in parallel and combines their outputs to capture different relationships in the same sequence.

How it is used
Deploys parallel causal attention heads inside transformer blocks to analyze relationships among sequence elements.
How deep it goes
Decomposes heads, learned projections, concatenation, tensor shapes, and wrapper code, then notes batched multiplication without implementing it.
Examples
Demonstrates two heads producing four features per token from a two-sequence, six-token batch and supplies a dimensionality exercise.
What you can do after
Builds multi-head attention with PyTorch modules, verifies output shapes, and places it inside a GPT transformer block.

Read how Build a Large Language Model (From Scratch) explains multi-head attention →

Deep Learning with R, Third Edition

MultiHeadAttention is a Keras layer that computes attention through multiple parallel attention heads and combines their outputs.

How it is used
Uses parallel attention heads to preserve distinct source-token relationships in sequence models and improve handling of long-range dependencies.
How deep it goes
Explains projections, scaled scores, softmax weighting, concatenation, and encoder integration, without deriving the underlying mathematics.
Examples
Uses R/Keras listings, source-target sequence attention, and the contextual example of “station” attending to “train.”
What you can do after
Build it from scratch or operate a Keras layer for cross-sequence attention, self-attention, and Transformer encoder blocks.

Read how Deep Learning with R, Third Edition explains multi-head attention →

Time Series Forecasting Using Foundation Models

An attention mechanism with multiple parallel attention heads that combines information from different representation subspaces.

How it is used
Uses parallel attention heads to learn diverse temporal relationships inside transformer-based forecasting models.
How deep it goes
Explains roles, masking, and architecture connections, but defers the mechanism's mathematical details.
Examples
Illustrates trend and seasonal heads, nearer historical tokens, masked decoder inputs, and TimesFM input patches.
What you can do after
Evaluate/understand how attention heads, causal masks, and encoder representations support transformer forecasts.

Read how Time Series Forecasting Using Foundation Models explains multi-head attention →

Machine Learning Algorithms in Depth

Multihead attention runs multiple attention mechanisms in parallel and combines their output representations.

How it is used
Builds transformer encoder–decoder and classification architectures from parallel attention representations.
How deep it goes
Defines parallel heads, concatenation, and dense transformation, but does not derive attention equations or head computations.
Examples
Illustrates translation attention, transformer architecture, multihead-attention structure, and Keras text classification.
What you can do after
Apply multihead attention through Keras in a transformer text classifier.

Read how Machine Learning Algorithms in Depth explains multi-head attention →

Transfer Learning for Natural Language Processing

Multihead attention is an attention mechanism that computes multiple representations of relationships within an input sequence.

How it is used
Refines transformer self-attention into a multidimensional mechanism associated with improved performance.
How deep it goes
Defines multi-head attention as a refinement of self-attention and visualizes its heads, without detailing their implementation.
Examples
Visualizes BERT attention in which “cells” is associated with “it” and “boring.”
What you can do after
Evaluate/understand attention visualizations by comparing shading across heads.

Read how Transfer Learning for Natural Language Processing explains multi-head attention →

Deep Learning with PyTorch, Second Edition

An attention mechanism that runs several attention heads in parallel, each learning different projections and relationships.

How it is used
Uses parallel attention heads inside transformer decoder and encoder-decoder architectures to capture diverse sequence relationships.
How deep it goes
Explains learned projections, head specialization, concatenation, and surrounding components, while stopping short of detailed attention-score mathematics.
Examples
Demonstrates a PyTorch TransformerBlock and a stacked character-level Transformer that generates sample names.
What you can do after
Builds it from scratch in PyTorch as part of a transformer block and connects it to sequence generation.

Read how Deep Learning with PyTorch, Second Edition explains multi-head attention →

Optimization Algorithms: AI techniques for design, planning, and control problems

Multi-head self-attention computes several attention operations in parallel so a sequence representation can capture different relationships among its elements.

How it is used
Uses parallel attention focuses to capture different relationships in graph neighborhoods and Transformer models.
How deep it goes
Explains multiple heads, neighborhood attention, softmax normalization, averaging, and final nonlinearities without implementation details.
Examples
Illustrates three attention heads aggregating features for node 5 before averaging their embeddings.
What you can do after
Evaluate how multiple attention heads combine distinct neighborhood perspectives into a prediction-ready representation.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains multi-head attention →

The AI Pocket Book

A transformer attention mechanism that splits representations into multiple heads to attend to different relationships in parallel.

How it is used
Uses parallel attention heads to improve contextualized representations during transformer-based language modeling.
How deep it goes
Defines separate vector chunks and explains their specialization, but stops short of detailing attention calculations or head outputs.
Examples
Illustrates specialization with hypothetical animal-related and tree-related embedding segments.
What you can do after
Evaluate how separate attention heads can specialize representations and outperform a single whole-vector attention head.

Read how The AI Pocket Book explains multi-head attention →

Related concepts

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.