liveBook · Manning

Attention matrix

An attention matrix is a matrix of scores or weights relating query positions to key positions and indicating their attention relationships.

How 5 Manning books explain it

Natural Language Processing in Action, Second Edition

An attention matrix is a matrix of learned or computed weights representing how strongly elements in a sequence should attend to one another.

How it is used
Uses attention matrices to connect token patterns across a sequence in transformer layers.
How deep it goes
Explains sequence-wide multiplication and causal masking without a worked calculation of attention weights.
Examples
Illustrates sequence encoding with [What] [is] [BERT] [?] and causal training with GPT-2.
What you can do after
Evaluate and understand how attention connects tokens and why decoder training masks future positions.

Read how Natural Language Processing in Action, Second Edition explains attention matrix →

Build a Text-to-Image Generator (from Scratch)

An attention map is a matrix of attention probabilities showing how strongly tokens attend to one another.

How it is used
Uses token-to-token probabilities to inspect what a vision transformer attends to while classifying images.
How deep it goes
Explains tensor dimensions, normalization, token selection, aggregation, reshaping, and interpolation, but not attention-score computation.
Examples
Uses CIFAR-10 images, 65-token matrices, 8 × 8 patch grids, and overlays comparing actual and predicted labels.
What you can do after
Diagnose attention behavior with PyTorch by extracting, validating, aggregating, reshaping, and visualizing attention maps.

Read how Build a Text-to-Image Generator (from Scratch) explains attention matrix →

Machine Learning System Design

An attention matrix is a two-dimensional array of attention weights relating positions in one sequence to positions in another sequence.

How it is used
Uses attention plots to validate sequence alignment assumptions, guide training, and locate errors for data-gathering decisions.
How deep it goes
Explains diagonal structure, residual interpretation, and guided-attention loss, but defers attention-module construction details.
Examples
Uses audio-frame-to-character plots, diagonal alignment, training evolution, and residuals around difficult sounds or character combinations.
What you can do after
Diagnose alignment problems with attention plots and use residual patterns to inform targeted data collection.

Read how Machine Learning System Design explains attention matrix →

Machines that Think

A self-attention matrix is a matrix of attention weights describing token-to-token attention across a sequence.

How it is used
Explains how self-attention distributes token relevance and combines sequence information into contextual representations.
How deep it goes
Traces relevance scores through row-wise softmax weights and value-vector combination to updated hidden states.
Examples
Uses “the sun rises” to illustrate stronger attention from “rises” to “sun” than to “the.”
What you can do after
Understand how token relevance weights control the information combined into contextual representations.

Read how Machines that Think explains attention matrix →

CUDA for LLMs

An attention probability matrix stores normalized attention weights for pairs of sequence positions.

How it is used
Identifies the memory bottleneck that motivates Flash Attention for long-context self-attention.
How deep it goes
Defines the QKᵀ-to-softmax pipeline, gives matrix sizes, and contrasts materialization with tiled processing.
Examples
Uses 8,192- and 128K-token contexts to quantify the memory required by an N × N matrix.
What you can do after
Understand why standard attention’s intermediate matrices create memory traffic and how Flash Attention avoids materializing the full matrix.

Read how CUDA for LLMs explains attention matrix →

What to know first

Related concepts

Questions about attention matrix

What is attention matrix?

An attention matrix is a matrix of scores or weights relating query positions to key positions and indicating their attention relationships.

Which Manning books explain attention matrix?

Natural Language Processing in Action, Second Edition, Build a Text-to-Image Generator (from Scratch), Machine Learning System Design, Machines that Think and CUDA for LLMs.

What should I know before learning attention matrix?

Manning books that teach attention matrix usually introduce attention, matrix-matrix multiplication and softmax first.

What is attention matrix a kind of?

Attention matrix is a kind of intermediate matrix. An intermediate matrix is a matrix produced between stages of a computation and consumed by later stages.

What are the parts of attention matrix?

Attention matrix includes attention weight.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.