liveBook · Manning

Value

As explained in Timeless Algorithms, The Foundational Ideas

A projected representation of information that is combined with other values according to attention weights.

In attention, a value is a representation of information that can contribute to another token’s contextual representation. Attention weights determine how much each value contributes: the output is a weighted combination of the values, rather than a selection based on a fixed grammatical rule. section 12.2.1

Queries and keys determine attention scores; the resulting weights are then applied to values. The distinction matters: scores describe how strongly tokens relate, while values supply the information being combined. In a simplified calculation, the query, key, and value are each set equal to the token embedding: q = k = v = e. section 12.2.1

Combining values for a pronoun

Suppose the values for “trophy” and “box” are [1, 0] and [0, 1], and the attention weights for “it” are 0.39 and 0.61. Its resulting representation is z_it = 0.39[1, 0] + 0.61[0, 1] = [0.39, 0.61]. The larger weight gives the box value more influence. These two-dimensional numbers are illustrative, not outputs from a trained model. section 12.2.1

The contextual representation contributes to predicting a probability distribution over possible next tokens. Real transformers use larger representations, multiple attention heads, feed-forward networks, and stacked layers, so the example isolates only part of that process. section 12.2.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Works through a two-dimensional weighted combination, while noting that real transformers use larger representations and additional components.
Examples
Calculates a representation for “it” from weighted trophy and box vectors.
What you can do after
Evaluate/understand how attention weights change the contribution of each value to a contextual representation.
How it connects
Connects values to queries, keys, embeddings, attention weights, and next-token prediction.

Also explained in

Compare how 15 books explain value →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.