Key

As explained in Deep Learning with PyTorch, Second Edition

The attention component that represents what a position contains and can be matched against a query.

A key represents what each position contains that might match a query. Attention compares queries with keys to decide which values deserve more weight, much like matching search text against indexed pages. section 9.5

In dot-product self-attention, learned transformations produce query and key vectors. Their dot products become attention scores; softmax normalizes those scores into weights that determine how strongly each value contributes to the output. section 9.5.1

Computing attention from query and key vectors

For an input x with shape [2, 3], create query, key, and value matrices with linear transformations. Compute q @ k.T, apply softmax across the scores, and multiply the resulting weights by v. Positions whose queries match keys more closely contribute more strongly through their values. section 9.5.1

Figure 9.7 Self attention, step-by-step from input to output

Follow the step-by-step flow from input positions through queries, keys, attention weights, and the output. section 9.5.1

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes

Explains key vectors, dot-product scores, softmax weights, weighted values, and a matrix-based implementation.

Examples

Uses next-character prediction, search-result retrieval, and a PyTorch tensor example with shape [2, 3].

What it assumes

Assumes familiarity with queries and values only at the level of their attention roles, while defining all three components together.

What you can do after

Apply it via dot-product attention to route more computation toward relevant positions.

How it connects

Connects keys with queries, values, attention, self-attention, cross-attention, dot products, and softmax.

liveBookmanning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Transformers in ActionIn attention, a key is a vector representation whose similarity to a query determines the relevance of its associated value.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A key is an attention vector representing information against which a query is compared to determine relevance.
• 

  liveBookmanning.com
  Sutskever's ListAn attention projection representing the information a sequence position offers for comparison with queries.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA key vector is an attention representation used to compute compatibility with a query.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchThe attention input that represents the searchable features used to match against a query.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)The attention-side vector associated with an input token and compared with a query to determine relevance.
• 

  liveBookmanning.com
  How Large Language Models WorkAn attention key is a vector representing the attributes against which an attention query is matched.

Compare how 8 books explain key →

Related concepts

• attentionused by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• grouped-query attentionused by · Grouped-query attention partitions query heads into groups whose members share key and value projections, reducing attention computation and memory use.
• Rotary Position Embeddingused by · Rotary Position Embedding is a positional encoding method that represents token positions by applying position-dependent rotations to components of attention query and key vectors.
• inner productuses · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
• key-value cachepart of · A key-value cache is a data structure that stores previously computed transformer attention keys and values for reuse during subsequent token generation.
• queryused by · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.