Key

As explained in Learn Generative AI with PyTorch

The attention input that represents the searchable features used to match against a query.

A key is the attention input that represents searchable features for matching against a query. In self-attention, the model projects input embeddings through a neural layer with weights WK to produce key vectors K. section 9.1.1

Attention compares query and key vectors with a dot product. Larger matching scores produce larger attention weights after scaling and softmax, while the value vectors provide the information combined into the attention output. section 9.1.1 section 1.3.1

Within scaled dot-product attention, the key participates in this sequence:

• Project the input embedding through the WK neural layer to obtain K. section 9.1.1
• Compare Q with K using a dot product, then divide by the square root of the key dimension, dk. section 9.1.1
• Apply softmax to the scaled scores to obtain attention weights that describe relationships across sequence elements. section 9.1.1 section 1.3.1

Library search analogy

For a search such as “machine learning in finance,” the query is the search phrase, while book titles and descriptions act as keys. Items whose keys match the query receive higher scores; the retrieved books correspond to values. section 1.3.1

Figure 9.1 A diagram of the self-attention mechanism. To calculate attention, the input embedding X is first passed through three neural layers with weights, W^Q, W^K, and W^V, respectively. The outputs are query Q, key K, and value V. The scaled attention score is the product of Q and K divided by the square root of the dimension of K, d_k. We apply the softmax function on the scaled attention score to obtain the attention weight. The attention is the product of the attention weight and value V.

Use the diagram to trace how input embedding X becomes query Q, key K, and value V, then produces scaled scores, attention weights, and the final attention output. section 9.1.1

How Learn Generative AI with PyTorch covers it

How deep it goes

Explains key construction, scaled query-key scoring, softmax weighting, and value combination without deriving the underlying neural layer.

Examples

Uses library search, English-to-French translation, token embeddings, and attention diagrams to illustrate keys.

What it assumes

Assumes familiarity with tokens and embeddings while introducing positional encoding and query-key-value attention in sequence.

What you can do after

Build it from scratch by implementing query-key-value attention for an English-to-French Transformer.

How it connects

Connects keys to queries, values, self-attention, scaled dot-product attention, positional encoding, and retrieval systems.

liveBookmanning.com

Read Learn Generative AI with PyTorch on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Transformers in ActionIn attention, a key is a vector representation whose similarity to a query determines the relevance of its associated value.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A key is an attention vector representing information against which a query is compared to determine relevance.
• 

  liveBookmanning.com
  Sutskever's ListAn attention projection representing the information a sequence position offers for comparison with queries.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA key vector is an attention representation used to compute compatibility with a query.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionThe attention component that represents what a position contains and can be matched against a query.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)The attention-side vector associated with an input token and compared with a query to determine relevance.
• 

  liveBookmanning.com
  How Large Language Models WorkAn attention key is a vector representing the attributes against which an attention query is matched.

Compare how 8 books explain key →

Related concepts

• attentionused by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• grouped-query attentionused by · Grouped-query attention partitions query heads into groups whose members share key and value projections, reducing attention computation and memory use.
• Rotary Position Embeddingused by · Rotary Position Embedding is a positional encoding method that represents token positions by applying position-dependent rotations to components of attention query and key vectors.
• inner productuses · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
• key-value cachepart of · A key-value cache is a data structure that stores previously computed transformer attention keys and values for reuse during subsequent token generation.
• queryused by · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.

This explainer was generated by Manning's knowledge graph from the text of Learn Generative AI with PyTorch. The section links point to the parts of the book it draws on.