Attention matrix
An attention matrix is a matrix of scores or weights indicating how strongly query positions attend to key positions.
Natural Language Processing in Action, Second Edition
An attention matrix is a matrix of learned or computed weights representing how strongly elements in a sequence should attend to one another.
- How it is used
- Connects tokens across a sequence while preserving stackable transformer layers and restricting what a text-generation decoder may see.
- How deep it goes
- Explains sequence-wide multiplication, encoding shapes, and causal masking, but does not derive how attention weights are calculated.
- Examples
- Uses a four-token question, an encoder word-connection diagram, and causal GPT-2 fine-tuning data.
- What you can do after
- Evaluate/understand it by tracing which tokens an encoder can connect and why a decoder must block future tokens.
Read how Natural Language Processing in Action, Second Edition explains attention matrix →
Build a Text-to-Image Generator (from Scratch)
An attention map is a matrix of attention probabilities showing how strongly tokens attend to one another.
- How it is used
- Uses token-to-token probabilities to inspect what a vision transformer attends to while classifying images.
- How deep it goes
- Explains tensor dimensions, normalization, token selection, aggregation, reshaping, and interpolation, but not attention-score computation.
- Examples
- Uses CIFAR-10 images, 65-token matrices, 8 × 8 patch grids, and overlays comparing actual and predicted labels.
- What you can do after
- Diagnose attention behavior with PyTorch by extracting, validating, aggregating, reshaping, and visualizing attention maps.
Read how Build a Text-to-Image Generator (from Scratch) explains attention matrix →
Machine Learning System Design
An attention matrix is a two-dimensional array of attention weights relating positions in one sequence to positions in another sequence.
- How it is used
- Uses attention plots to validate sequence alignment assumptions, guide training, and locate errors for data-gathering decisions.
- How deep it goes
- Explains diagonal structure, residual interpretation, and guided-attention loss, but defers attention-module construction details.
- Examples
- Uses audio-frame-to-character plots, diagonal alignment, training evolution, and residuals around difficult sounds or character combinations.
- What you can do after
- Diagnose alignment problems with attention plots and use residual patterns to inform targeted data collection.
Read how Machine Learning System Design explains attention matrix →
Related concepts
- inner productproduced by · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
- Vision Transformerproduced by · A Vision Transformer is a Transformer-based neural-network architecture that processes images as sequences of tokenized regions using attention.
- attention weighthas part · An attention weight is a computed coefficient, often normalized, that determines how strongly an input element or value contributes to an attention-based representation.
- attention headproduced by · An attention head is an independently parameterized attention computation that evaluates relationships among input elements and combines information from them into a contextualized representation.
- attention plotmeasured by · An attention plot is a visualization of attention activations between elements of an input sequence and elements of an output sequence.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.