liveBook · Manning

Attention weight

As explained in Machines that Think

An attention weight is a normalized value indicating how much one token attends to another token.

An attention weight is a normalized value showing how much relative attention one token gives to each token in the sequence. It is produced by applying softmax to relevance scores, so the weights in each row sum to one. section 4.1.3

The weights determine how much each token’s value vector contributes to another token’s updated representation. A higher weight means a greater contribution; the resulting context is incorporated into hidden states. section 4.1.3

“Rises” attends to “sun”

If “sun” receives a higher weight than “the” for “rises,” more information from “sun” contributes to the updated representation of “rises.” section 4.1.3

Figure 4.5 The model turns each token’s representation into three vectors: a query that expresses what it seeks, a key that describes characteristics used to assess relevance, and a value that carries the information it can contribute. Each is produced by its own learned projection. By comparing queries and keys, the attention head produces an attention pattern that reflects how strongly tokens are related in the model’s internal representation, with darker cells in the matrix indicating stronger connections. The values are then combined according to this pattern, giving each token a new representation, known as a hidden state, that integrates contextual information from across the sequence.

The diagram connects query-key comparisons to the attention pattern and shows how weighted values form contextual representations. section 4.1.3

How Machines that Think covers it

How deep it goes
Explains row-wise softmax normalization and weighted value combination as parts of a single attention head.
Examples
Uses “rises” attending more to “sun” than “the” to illustrate weighted value contribution.
What you can do after
Understand how normalized weights guide value-vector contributions to contextual token representations.
How it connects
Connects attention weights to relevance scores, softmax, value vectors, self-attention matrices, and hidden states.

Also explained in

Compare how 7 books explain attention weight →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.