Attention matrix
An attention matrix is a matrix of scores or weights relating query positions to key positions and indicating their attention relationships.
Natural Language Processing in Action, Second Edition
An attention matrix is a matrix of learned or computed weights representing how strongly elements in a sequence should attend to one another.
- How it is used
- Uses attention matrices to connect token patterns across a sequence in transformer layers.
- How deep it goes
- Explains sequence-wide multiplication and causal masking without a worked calculation of attention weights.
- Examples
- Illustrates sequence encoding with [What] [is] [BERT] [?] and causal training with GPT-2.
- What you can do after
- Evaluate and understand how attention connects tokens and why decoder training masks future positions.
Read how Natural Language Processing in Action, Second Edition explains attention matrix →
Build a Text-to-Image Generator (from Scratch)
An attention map is a matrix of attention probabilities showing how strongly tokens attend to one another.
- How it is used
- Uses token-to-token probabilities to inspect what a vision transformer attends to while classifying images.
- How deep it goes
- Explains tensor dimensions, normalization, token selection, aggregation, reshaping, and interpolation, but not attention-score computation.
- Examples
- Uses CIFAR-10 images, 65-token matrices, 8 × 8 patch grids, and overlays comparing actual and predicted labels.
- What you can do after
- Diagnose attention behavior with PyTorch by extracting, validating, aggregating, reshaping, and visualizing attention maps.
Read how Build a Text-to-Image Generator (from Scratch) explains attention matrix →
Machine Learning System Design
An attention matrix is a two-dimensional array of attention weights relating positions in one sequence to positions in another sequence.
- How it is used
- Uses attention plots to validate sequence alignment assumptions, guide training, and locate errors for data-gathering decisions.
- How deep it goes
- Explains diagonal structure, residual interpretation, and guided-attention loss, but defers attention-module construction details.
- Examples
- Uses audio-frame-to-character plots, diagonal alignment, training evolution, and residuals around difficult sounds or character combinations.
- What you can do after
- Diagnose alignment problems with attention plots and use residual patterns to inform targeted data collection.
Read how Machine Learning System Design explains attention matrix →
Machines that Think
A self-attention matrix is a matrix of attention weights describing token-to-token attention across a sequence.
- How it is used
- Explains how self-attention distributes token relevance and combines sequence information into contextual representations.
- How deep it goes
- Traces relevance scores through row-wise softmax weights and value-vector combination to updated hidden states.
- Examples
- Uses “the sun rises” to illustrate stronger attention from “rises” to “sun” than to “the.”
- What you can do after
- Understand how token relevance weights control the information combined into contextual representations.
CUDA for LLMs
An attention probability matrix stores normalized attention weights for pairs of sequence positions.
- How it is used
- Identifies the memory bottleneck that motivates Flash Attention for long-context self-attention.
- How deep it goes
- Defines the QKᵀ-to-softmax pipeline, gives matrix sizes, and contrasts materialization with tiled processing.
- Examples
- Uses 8,192- and 128K-token contexts to quantify the memory required by an N × N matrix.
- What you can do after
- Understand why standard attention’s intermediate matrices create memory traffic and how Flash Attention avoids materializing the full matrix.
What to know first
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- softmaxSoftmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
Related concepts
- inner productproduced by · An inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
- self-attentionproduced by · Self-attention is an attention mechanism that relates positions within a sequence, allowing each position to incorporate information from other positions in that sequence.
- attention weighthas part · An attention weight is a coefficient indicating how strongly an associated input or value contributes to an attention-based output or representation.
- Vision Transformerproduced by · A Vision Transformer is a Transformer-based neural-network architecture that processes images as sequences of tokenized regions using attention.
- attention headproduced by · An attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.
Questions about attention matrix
What is attention matrix?
An attention matrix is a matrix of scores or weights relating query positions to key positions and indicating their attention relationships.
Which Manning books explain attention matrix?
Natural Language Processing in Action, Second Edition, Build a Text-to-Image Generator (from Scratch), Machine Learning System Design, Machines that Think and CUDA for LLMs.
What should I know before learning attention matrix?
Manning books that teach attention matrix usually introduce attention, matrix-matrix multiplication and softmax first.
What is attention matrix a kind of?
Attention matrix is a kind of intermediate matrix. An intermediate matrix is a matrix produced between stages of a computation and consumed by later stages.
What are the parts of attention matrix?
Attention matrix includes attention weight.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




