liveBook · Manning

Key

As explained in Timeless Algorithms, The Foundational Ideas

In an attention mechanism, a projected representation of an element that is compared with queries to determine relevance.

When a model encounters an ambiguous pronoun, it can compare that pronoun with surrounding tokens to build a contextual representation. In self-attention, a key is the projected representation through which a token signals its relevance to another token’s query. A value holds the information the token can contribute if attention assigns it weight. section 12.2.1 section 12.2

Each token’s embedding is transformed by a learned key projection, W_K. Separate projections produce its query and value, allowing the same token to be represented differently for seeking information, signaling relevance, and contributing content. A query–key dot product measures alignment; the resulting scores are scaled and passed through softmax to obtain attention weights. Those weights determine how values are combined. section 12.2.1

Comparing a pronoun with two keys

Suppose the query for “it” is [0.2, 0.8], and the keys for “trophy” and “box” are [1, 0] and [0, 1]. Their dot products with the query are 0.2 and 0.8. After scaling and softmax, the approximate weights are 0.39 and 0.61, so the box’s value contributes more to the contextual representation. The projections are assumed to leave the vectors unchanged; the illustrative coordinates are not literal probabilities or typical interpretable dimensions of a trained model. section 12.2.1

A single attention calculation does not account for the full behavior of a trained transformer. Many attention heads, feed-forward networks, and stacked layers operate together. section 12.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Works through projection, dot products, scaling, softmax, and weighted values numerically, while simplifying the behavior of a complete transformer.
Examples
Uses pronoun resolution with a trophy and box, alongside semantic association, factual recall, and autoregressive generation.
What it assumes
Introduces dot products through two-dimensional vectors and draws on the idea of learned weights.
What you can do after
Evaluate how query–key alignment affects attention weights and contextual representations in a simplified transformer.
How it connects
Connects keys with queries, values, embeddings, dot products, softmax, and self-attention.

Also explained in

Compare how 15 books explain key →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.