Key
In attention, a key is a vector representation of an input element that is compared with a query to determine relevance and attention weights for its associated value.
Transformers in Action
In attention, a key is a vector representation whose similarity to a query determines the relevance of its associated value.
- How it is used
- Determines how strongly each value contributes to an attention output by matching keys against queries.
- How deep it goes
- Defines the key’s role and works through its projected matrix in scaled dot-product and multihead attention implementations.
- Examples
- Uses “The movie was not bad” and a NumPy attention listing to show matching and weight calculation.
- What you can do after
- Build it from scratch in a simplified NumPy attention calculation using projected keys, scaled query–key scores, and weighted values.
Build a Text-to-Image Generator (from Scratch)
A key is an attention vector representing information against which a query is compared to determine relevance.
- How it is used
- Uses keys to compare sequence elements and determine which values receive attention.
- How deep it goes
- Explains key projections, dot-product scoring, scaling, softmax, dimensions, and multi-head implementation.
- Examples
- Uses German-to-English translation, 48-dimensional vision tokens split into four heads, and a PyTorch attention-head listing.
- What you can do after
- Builds key projections and scaled query–key attention with PyTorch linear layers and softmax.
Read how Build a Text-to-Image Generator (from Scratch) explains key →
Sutskever's List
An attention projection representing the information a sequence position offers for comparison with queries.
- How it is used
- Represents encoder-side information that decoder queries compare against during cross-attention.
- How deep it goes
- Defines keys through their attention role and illustrates their interaction with queries, without deriving projection internals.
- Examples
- Uses a Transformer architecture schematic showing decoder queries interacting with encoder keys and values.
- What you can do after
- Understand how keys support relevance scoring and context construction during Transformer cross-attention.
Deep Learning with R, Third Edition
A key vector is an attention representation used to compute compatibility with a query.
- How it is used
- Uses key vectors to score source information against queries and to cache past context during causal text generation.
- How deep it goes
- Explains key vectors through lookup, projection, dot-product scoring, causal masking, and caching, without deriving attention mathematically.
- Examples
- Illustrates key vectors through photo-tag retrieval, parameterized attention code, and cached key-value vectors in mini-GPT generation.
- What you can do after
- Apply key vectors via dot-product attention and implement key-value caching for efficient causal generation.
Learn Generative AI with PyTorch
The attention input that represents the searchable features used to match against a query.
- How it is used
- Uses searchable features to determine how strongly each sequence element relates to a query.
- How deep it goes
- Explains key construction, scaled query-key scoring, softmax weighting, and value combination without deriving the underlying neural layer.
- Examples
- Uses library search, English-to-French translation, token embeddings, and attention diagrams to illustrate keys.
- What you can do after
- Build it from scratch by implementing query-key-value attention for an English-to-French Transformer.
Deep Learning with PyTorch, Second Edition
The attention component that represents what a position contains and can be matched against a query.
- How it is used
- Matches each position’s contents against a query so attention can select relevant information.
- How deep it goes
- Explains key vectors, dot-product scores, softmax weights, weighted values, and a matrix-based implementation.
- Examples
- Uses next-character prediction, search-result retrieval, and a PyTorch tensor example with shape [2, 3].
- What you can do after
- Apply it via dot-product attention to route more computation toward relevant positions.
Read how Deep Learning with PyTorch, Second Edition explains key →
Build a Large Language Model (From Scratch)
The attention-side vector associated with an input token and compared with a query to determine relevance.
- How it is used
- Uses keys in self-attention to compare transformed input tokens with a query and determine relevance.
- How deep it goes
- Traces key projection, tensor shapes, dot-product scoring, scaling, softmax, and value aggregation.
- Examples
- Uses a six-token sentence, a second-token query, W_key projection, and [6, 2] key tensors.
- What you can do after
- Build key tensors in PyTorch and use them in scaled dot-product attention computations.
Read how Build a Large Language Model (From Scratch) explains key →
How Large Language Models Work
An attention key is a vector representing the attributes against which an attention query is matched.
- How it is used
- Matches query vectors against possible answers within transformer attention.
- How deep it goes
- Defines keys, relates them to values and queries, and stops short of detailed vector calculations.
- Examples
- Uses Python dictionary lookup and preceding words in next-token prediction as concrete examples.
- What you can do after
- Understand how attention uses similarity-weighted keys to select contextual information.
Related concepts
- attentionused by · Attention is a neural-network mechanism that computes input-dependent weights over elements and uses them to combine information when forming representations or outputs.
- grouped-query attentionused by · Grouped-query attention partitions query heads into groups whose members share key and value projections, reducing attention computation and memory use.
- Rotary Position Embeddingused by · Rotary Position Embedding is a positional encoding method that applies position-dependent rotations to attention query and key components to represent sequence positions.
- inner productuses · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
- key-value cachepart of · A key-value cache is a data structure that stores previously computed transformer attention keys and values for reuse during subsequent token generation.
- queryused by · A query is a representation of information sought, used to identify or evaluate relevant items or other representations.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.