liveBook · Manning

Attention matrix

As explained in Machines that Think

A self-attention matrix is a matrix of attention weights describing token-to-token attention across a sequence.

A self-attention matrix arranges token-to-token attention weights across a sequence. Each row gives one token’s relative attention to the others: softmax converts its relevance scores into weights that sum to one. These weights are then applied to value vectors, determining how much information each token contributes to updated, contextual representations. section 4.1.3

“The sun rises”

For “rises,” a higher weight on “sun” than on “the” means more information from “sun” contributes to its updated representation. section 4.1.3

Figure 4.5 The model turns each token’s representation into three vectors: a query that expresses what it seeks, a key that describes characteristics used to assess relevance, and a value that carries the information it can contribute. Each is produced by its own learned projection. By comparing queries and keys, the attention head produces an attention pattern that reflects how strongly tokens are related in the model’s internal representation, with darker cells in the matrix indicating stronger connections. The values are then combined according to this pattern, giving each token a new representation, known as a hidden state, that integrates contextual information from across the sequence.

The figure shows how query-key comparisons produce the attention pattern and how weighted values form contextualized hidden states. section 4.1.3

How Machines that Think covers it

How deep it goes
Traces relevance scores through row-wise softmax weights and value-vector combination to updated hidden states.
Examples
Uses “the sun rises” to illustrate stronger attention from “rises” to “sun” than to “the.”
What you can do after
Understand how token relevance weights control the information combined into contextual representations.
How it connects
Connects attention weights to queries, keys, value vectors, attention heads, and hidden states.

Also explained in

Compare how 5 books explain attention matrix →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.