liveBook · Manning

Multi-head attention

Also called multihead attention, multiheaded attention, MHA, MultiHeadAttention

Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.

How 25 Manning books explain it

Natural Language Processing in Action, Second Edition

Multi-head attention computes multiple attention representations in parallel and combines them, potentially using distinct query and source sequences.

How it is used
Uses parallel attention heads to form transformer representations for language and translation.
How deep it goes
Explains scaled attention, head concatenation, output projection, and decoder customization rather than implementing attention from scratch.
Examples
Uses an eight-head transformer configuration and German–English translation with visualized decoder attention weights.
What you can do after
Apply it via PyTorch by customizing a transformer decoder to retain and inspect attention weights.

Read how Natural Language Processing in Action, Second Edition explains multi-head attention →

Build a Text-to-Image Generator (from Scratch)

An attention mechanism that computes attention through multiple separate query, key, and value subspaces to capture different relationships simultaneously.

How it is used
Uses multiple attention heads to capture different relationships in text and image inputs.
How deep it goes
Works through per-head projections and scaled dot-product calculations, but leaves one text transformer's full MultiHeadedAttention class definition off-page.
Examples
Illustrates heads with an ambiguous “bank” pun and image features such as texture and edges.
What you can do after
Builds attention heads from scratch in PyTorch and integrates multi-head attention into transformer encoder blocks.

Read how Build a Text-to-Image Generator (from Scratch) explains multi-head attention →

Transformers in Action

Multihead attention is an attention mechanism that computes multiple attention patterns in parallel so a model can represent different relationships among sequence positions.

How it is used
Uses parallel attention patterns to capture multiple relationships and long-range dependencies in transformer sequences.
How deep it goes
Explains attention calculations and encoder placement, then illustrates projections, head dimensions, and a layer invocation.
Examples
Illustrates negation in sentiment classification and a 512-dimensional self-attention call with eight heads.
What you can do after
Apply it via a PyTorch-style multi-head layer to a batch of sequences, using the same input for queries, keys, and values.

Read how Transformers in Action explains multi-head attention →

Build a DeepSeek Model (From Scratch)

Multi-head self-attention computes attention over an input sequence through multiple parallel attention heads, with each head having its own learned projections.

How it is used
Uses multi-head attention as the Transformer baseline for analyzing next-token inference and KV cache bottlenecks.
How deep it goes
Works through projections, attention scores, masking, context vectors, head combination, and cache-memory scaling.
Examples
Uses a four-token projection walkthrough, an ambiguous brush sentence, and a four-architecture training comparison.
What you can do after
Evaluate attention designs by tracing head computations and comparing expressivity with KV cache memory.

Read how Build a DeepSeek Model (From Scratch) explains multi-head attention →

Build AI Drug Discovery Pipelines

Multi-head attention is an attention mechanism that combines multiple independently parameterized attention operations to capture different relationships in a sequence.

How it is used
Uses multiple attention heads to model protein-sequence relationships and weight molecular-graph neighbors for drug-target affinity prediction.
How deep it goes
Explains independent attention projections and output fusion, and demonstrates graph-layer configuration while deferring the full protein encoder implementation to accompanying resources.
Examples
Illustrates heads with protein attention heatmaps and a two-head molecular GATConv layer.
What you can do after
Apply it via GATConv by configuring the number of heads and choosing whether to concatenate or average their outputs.

Read how Build AI Drug Discovery Pipelines explains multi-head attention →

Learn Generative AI with PyTorch

A self-attention variant that splits representations into multiple attention heads to capture different relationships in parallel.

How it is used
Uses multi-head attention as a Transformer component for contextual representations in translation and causal text generation.
How deep it goes
Works through projections, head reshaping, scaled attention, masking, and output concatenation with tensor dimensions and PyTorch code.
Examples
Walks through an eight-head sentence representation and four-head attention over flattened image features.
What you can do after
Build it from scratch in PyTorch as part of a Transformer, using attention scores, masks, and head reshaping.

Read how Learn Generative AI with PyTorch explains multi-head attention →

Build an Advanced RAG Application (From Scratch)

A multi-head self-attention mechanism applies multiple attention operations in parallel so a model can capture different relationships among sequence elements.

How it is used
Supports Transformer sequence understanding by relating input positions and providing a comprehensive view of their relationships.
How deep it goes
Explains the parallel weighting intuition and long-range dependency role, without detailing head projections or attention calculations.
Examples
Illustrates parallel attention as many pairs of eyes following different characters and events in a story.
What you can do after
Evaluate/understand how Transformers capture varied and distant relationships across a sequence.

Read how Build an Advanced RAG Application (From Scratch) explains multi-head attention →

Applied Reinforcement Learning

Multi-head attention performs several independent attention computations in parallel and combines their outputs to capture different relationship patterns.

How it is used
Enriches zone representations for a reinforcement-learning policy by capturing several inter-zone relationship patterns at once.
How deep it goes
Details learned Query, Key, and Value projections, scaled dot-product attention, concatenation, and final linear projection.
Examples
Uses thermal proximity, workload correlation, shared cooling infrastructure, and pressure dynamics as inter-zone relationships.
What you can do after
Understand how parallel attention heads represent multiple kinds of relationships between zones.

Read how Applied Reinforcement Learning explains multi-head attention →

CUDA for LLMs

Multihead attention runs multiple attention heads and combines their outputs with a projection.

How it is used
Uses multi-head attention within decoder-only transformer blocks for autoregressive token prediction.
How deep it goes
Explains head computation and multi-head aggregation through PyTorch initialization and forward passes.
Examples
Uses a four-head, 128-channel transformer with 64-token sequences and eight blocks.
What you can do after
Build it from scratch in PyTorch using per-head modules, concatenation, and a linear projection.

Read how CUDA for LLMs explains multi-head attention →

Inside Deep Learning: Math, Algorithms, Models

An attention mechanism that uses multiple attention heads to attend to different contexts or aspects of the input in parallel.

How it is used
Extends attention inside the vanilla transformer to detect multiple input patterns in parallel.
How deep it goes
Defines query, key, value, and output shapes, then sketches per-head projections and concatenation without detailing the full attention calculation.
Examples
Illustrates pattern detection with “not” preceding “good” and compares attention queries with Python dictionary lookups.
What you can do after
Build a multi-head layer by projecting queries, keys, and values per head, concatenating responses, and mapping them back to D dimensions.

Read how Inside Deep Learning: Math, Algorithms, Models explains multi-head attention →

Deep Learning with Python, Third Edition

An attention variant that runs several attention computations in parallel with different parameters to learn different relationships.

How it is used
Uses multiple attention heads to enrich sequence representations in Transformer encoders and decoders.
How deep it goes
Works through projections, attention scores, head concatenation, and Keras encoder and decoder applications.
Examples
Illustrates heads attending to sentence subjects and punctuation, and self-attention relating “station” to “train.”
What you can do after
Apply it via Keras MultiHeadAttention for encoder self-attention and masked decoder attention.

Read how Deep Learning with Python, Third Edition explains multi-head attention →

Graph Neural Networks in Action

An attention technique that computes multiple attention operations in parallel and combines their outputs.

How it is used
Uses multiple attention heads in GAT layers to learn which neighboring nodes matter when updating node features.
How deep it goes
Explains neighbor attention scores and aggregation, then shows head-dependent layer dimensions without a rigorous derivation.
Examples
Shows a GATConv architecture with two heads in its first layer and one head in its output layer.
What you can do after
Apply it via PyTorch Geometric GATConv by setting heads and matching subsequent layers to the resulting feature dimensions.

Read how Graph Neural Networks in Action explains multi-head attention →

Rearchitecting LLMs

Multi-head attention is an attention mechanism that computes attention through multiple parallel representation subspaces and combines their outputs.

How it is used
Explains parallel attention heads as a way to contextualize embeddings and motivates memory-efficient attention variants.
How deep it goes
Explains head dimensions and Q/K/V roles, then compares MHA, GQA, and MQA cache storage.
Examples
Uses DistilGPT2’s 12 heads over 768 dimensions, a “spring” token example, and cache-budget comparisons with GQA.
What you can do after
Evaluate how head partitioning contextualizes embeddings and how shared Key and Value pairs reduce cache use.

Read how Rearchitecting LLMs explains multi-head attention →

Build a Large Language Model (From Scratch)

An attention technique that computes several independent attention heads in parallel and combines their outputs to capture different relationships in the same sequence.

How it is used
Deploys parallel causal attention heads inside transformer blocks to analyze relationships among sequence elements.
How deep it goes
Decomposes heads, learned projections, concatenation, tensor shapes, and wrapper code, then notes batched multiplication without implementing it.
Examples
Demonstrates two heads producing four features per token from a two-sequence, six-token batch and supplies a dimensionality exercise.
What you can do after
Builds multi-head attention with PyTorch modules, verifies output shapes, and places it inside a GPT transformer block.

Read how Build a Large Language Model (From Scratch) explains multi-head attention →

Deep Learning with R, Third Edition

MultiHeadAttention is a Keras layer that computes attention through multiple parallel attention heads and combines their outputs.

How it is used
Uses parallel attention heads to preserve distinct source-token relationships in sequence models and improve handling of long-range dependencies.
How deep it goes
Explains projections, scaled scores, softmax weighting, concatenation, and encoder integration, without deriving the underlying mathematics.
Examples
Uses R/Keras listings, source-target sequence attention, and the contextual example of “station” attending to “train.”
What you can do after
Build it from scratch or operate a Keras layer for cross-sequence attention, self-attention, and Transformer encoder blocks.

Read how Deep Learning with R, Third Edition explains multi-head attention →

Time Series Forecasting Using Foundation Models

An attention mechanism with multiple parallel attention heads that combines information from different representation subspaces.

How it is used
Uses parallel attention heads to learn diverse temporal relationships inside transformer-based forecasting models.
How deep it goes
Explains roles, masking, and architecture connections, but defers the mechanism's mathematical details.
Examples
Illustrates trend and seasonal heads, nearer historical tokens, masked decoder inputs, and TimesFM input patches.
What you can do after
Evaluate/understand how attention heads, causal masks, and encoder representations support transformer forecasts.

Read how Time Series Forecasting Using Foundation Models explains multi-head attention →

Building Specialized AI Systems

An attention method that runs attention across multiple learned representation subspaces and concatenates the resulting outputs.

How it is used
Explains how Transformer layers capture multiple contextual relationships while processing input or generating output sequences.
How deep it goes
Explains QKV projections, scaled scores, softmax weights, value aggregation, and head concatenation, but stops short of implementation code.
Examples
Uses pronoun resolution, scaled dot-product attention, and an encoder-decoder Transformer architecture as concrete teaching examples.
What you can do after
Evaluate how parallel attention heads create context-aware representations within Transformer encoder and decoder blocks.

Read how Building Specialized AI Systems explains multi-head attention →

Timeless Algorithms, The Foundational Ideas

An attention mechanism that computes multiple attention patterns in parallel using distinct learned projections and combines their outputs.

How it is used
Expands self-attention’s relational capacity by letting several attention patterns contribute simultaneously.
How deep it goes
Explains learned projections, combined head outputs, possible head roles, and their limits of interpretability.
Examples
Illustrates translation relationships and code dependencies that can influence a token simultaneously.
What you can do after
Understand how learned attention heads combine multiple relational patterns rather than relying on one dominant relationship.

Read how Timeless Algorithms, The Foundational Ideas explains multi-head attention →

Machine Learning Algorithms in Depth

Multihead attention runs multiple attention mechanisms in parallel and combines their output representations.

How it is used
Builds transformer encoder–decoder and classification architectures from parallel attention representations.
How deep it goes
Defines parallel heads, concatenation, and dense transformation, but does not derive attention equations or head computations.
Examples
Illustrates translation attention, transformer architecture, multihead-attention structure, and Keras text classification.
What you can do after
Apply multihead attention through Keras in a transformer text classifier.

Read how Machine Learning Algorithms in Depth explains multi-head attention →

Transfer Learning for Natural Language Processing

Multihead attention is an attention mechanism that computes multiple representations of relationships within an input sequence.

How it is used
Refines transformer self-attention into a multidimensional mechanism associated with improved performance.
How deep it goes
Defines multi-head attention as a refinement of self-attention and visualizes its heads, without detailing their implementation.
Examples
Visualizes BERT attention in which “cells” is associated with “it” and “boring.”
What you can do after
Evaluate/understand attention visualizations by comparing shading across heads.

Read how Transfer Learning for Natural Language Processing explains multi-head attention →

Deep Learning with PyTorch, Second Edition

An attention mechanism that runs several attention heads in parallel, each learning different projections and relationships.

How it is used
Uses parallel attention heads inside transformer decoder and encoder-decoder architectures to capture diverse sequence relationships.
How deep it goes
Explains learned projections, head specialization, concatenation, and surrounding components, while stopping short of detailed attention-score mathematics.
Examples
Demonstrates a PyTorch TransformerBlock and a stacked character-level Transformer that generates sample names.
What you can do after
Builds it from scratch in PyTorch as part of a transformer block and connects it to sequence generation.

Read how Deep Learning with PyTorch, Second Edition explains multi-head attention →

Optimization Algorithms: AI techniques for design, planning, and control problems

Multi-head self-attention computes several attention operations in parallel so a sequence representation can capture different relationships among its elements.

How it is used
Uses parallel attention focuses to capture different relationships in graph neighborhoods and Transformer models.
How deep it goes
Explains multiple heads, neighborhood attention, softmax normalization, averaging, and final nonlinearities without implementation details.
Examples
Illustrates three attention heads aggregating features for node 5 before averaging their embeddings.
What you can do after
Evaluate how multiple attention heads combine distinct neighborhood perspectives into a prediction-ready representation.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains multi-head attention →

Fight Fraud with Machine Learning

Multi-head attention is an attention module containing multiple self-attention layers whose outputs are combined to form a richer representation.

How it is used
Explains how transformers extract complex information from sequential data for tasks including fraud detection.
How deep it goes
Defines heads, attention coefficients, and concatenation, but defers detailed transformer architecture.
Examples
Uses time-step sequences, attention weights Pq/Pk/Pv, coefficients c1–cn, and concatenated feature representations.
What you can do after
Evaluate how multiple attention heads capture different signals from a sequence without recurrent connections.

Read how Fight Fraud with Machine Learning explains multi-head attention →

Understanding AI

Multi-head attention is an attention mechanism that performs several attention operations in parallel and combines their results.

How it is used
Uses parallel attention processes to build context-aware token representations for coherent next-token generation.
How deep it goes
Explains parallel heads, learned specializations, and combined results, without deriving the underlying attention calculations.
Examples
Uses pronoun resolution for “it,” its link to “performance,” and the analogy of multiple readers highlighting one sentence.
What you can do after
Understand how parallel attention heads combine different learned relationships into context-aware representations.

Read how Understanding AI explains multi-head attention →

The AI Pocket Book

A transformer attention mechanism that splits representations into multiple heads to attend to different relationships in parallel.

How it is used
Uses parallel attention heads to improve contextualized representations during transformer-based language modeling.
How deep it goes
Defines separate vector chunks and explains their specialization, but stops short of detailing attention calculations or head outputs.
Examples
Illustrates specialization with hypothetical animal-related and tree-related embedding segments.
What you can do after
Evaluate how separate attention heads can specialize representations and outperform a single whole-vector attention head.

Read how The AI Pocket Book explains multi-head attention →

Also covered in

What to know first

Related concepts

Questions about multi-head attention

What is multi-head attention?

Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.

Which Manning books explain multi-head attention?

Natural Language Processing in Action, Second Edition, Build a Text-to-Image Generator (from Scratch), Transformers in Action, Build a DeepSeek Model (From Scratch), Build AI Drug Discovery Pipelines, Learn Generative AI with PyTorch, Build an Advanced RAG Application (From Scratch), Applied Reinforcement Learning and 17 more.

What should I know before learning multi-head attention?

Manning books that teach multi-head attention usually introduce attention, recurrent neural network, transformer architecture and softmax first.

What is multi-head attention a kind of?

Multi-head attention is a kind of attention. Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.

What is multi-head attention part of?

Multi-head attention is part of transformer block. A transformer block is a modular unit of a transformer neural network that applies attention and other transformations, commonly including feed-forward computation, to representations.

What are the parts of multi-head attention?

Multi-head attention includes attention head.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.