liveBook · Manning

Key

As explained in Machines that Think

A key is an attention vector representing the properties by which a sequence element can be matched to a query.

A key is a learned vector derived from a token’s representation. It describes characteristics that help a query assess whether that token is relevant. Unlike a key, a value carries information that can be added to another token’s representation. section 4.1.3

An attention layer compares a query with keys using a dot product: stronger alignment produces a higher relevance score. Keys can come from the same sequence as the query in self-attention, or from encoder representations in cross-attention. section 4.1.3 section 4.1.7

Matching “rises”

For “rises” in “The sun rises in the east,” its query can match the key for “sun” more strongly than the key for “the,” giving “sun” a higher relevance score. section 4.1.3

Figure 4.3 A single token representation is transformed by three learned projections into a query, key, and value, which serve different roles in attention. The query represents what the token looks for in context, the key what makes it potentially relevant to other tokens, and the value the information it can contribute.

The diagram distinguishes the key’s relevance-matching role from the query’s search role and the value’s information-carrying role. section 4.1.3

How Machines that Think covers it

How deep it goes
Defines keys as learned projections and explains query-key dot-product scoring, but does not detail how the projections are trained.
Examples
Illustrates query-key matching with “rises” and gives “sun” a stronger match than “the.”
What it assumes
Starts from token representations and distinguishes query, key, and value roles rather than assuming those roles are already known.
What you can do after
Understand how a query compares with keys to assign token relevance before values contribute information.
How it connects
Connects keys to queries and values in self-attention, and to encoder representations in cross-attention.

Also explained in

Compare how 15 books explain key →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.