Key

As explained in Build a Large Language Model (From Scratch)

The attention-side vector associated with an input token and compared with a query to determine relevance.

A key is the attention-side vector associated with an input token. Self-attention compares a query with keys to produce relevance scores, then uses those scores to combine value vectors. section 3.4.1 section 3.7

The implementation follows this data flow:

• Project each input embedding through the trainable matrix W_key: keys = inputs @ W_key. section 3.4.1
• Compare one query with every key using dot products to obtain attention scores. section 3.4.1
• Scale and normalize the scores with softmax; the resulting attention weights determine how strongly each value contributes. section 3.4.1

Six-token input

For the sentence “Your journey starts with one step,” a six-token input produces keys with shape [6, 2] when d_out = 2. A query for the second token compares against all six keys. section 3.4.1

Figure 3.14 In the first step of the self-attention mechanism with trainable weight matrices, we compute query (q), key (k), and value (v) vectors for input elements x. Similar to previous sections, we designate the second input, x⁽²⁾, as the query input. The query vector q⁽²⁾ is obtained via matrix multiplication between the input x⁽²⁾ and the weight matrix W_q. Similarly, we obtain the key and value vectors via matrix multiplication involving the weight matrices W_k and W_v.

Trace how an input element becomes a key through matrix multiplication alongside its query and value vectors. section 3.4.1

How Build a Large Language Model (From Scratch) covers it

How deep it goes

Traces key projection, tensor shapes, dot-product scoring, scaling, softmax, and value aggregation.

Examples

Uses a six-token sentence, a second-token query, W_key projection, and [6, 2] key tensors.

What it assumes

Starts from embedded input tokens and introduces the query, key, and value projections needed for self-attention.

What you can do after

Build key tensors in PyTorch and use them in scaled dot-product attention computations.

How it connects

Connects keys to queries, values, attention weights, scaled-dot product attention, causal attention masks, and multi-head attention.

liveBookmanning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Transformers in ActionIn attention, a key is a vector representation whose similarity to a query determines the relevance of its associated value.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A key is an attention vector representing information against which a query is compared to determine relevance.
• 

  liveBookmanning.com
  Sutskever's ListAn attention projection representing the information a sequence position offers for comparison with queries.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA key vector is an attention representation used to compute compatibility with a query.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchThe attention input that represents the searchable features used to match against a query.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionThe attention component that represents what a position contains and can be matched against a query.
• 

  liveBookmanning.com
  How Large Language Models WorkAn attention key is a vector representing the attributes against which an attention query is matched.

Compare how 8 books explain key →

Related concepts

• attentionused by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• grouped-query attentionused by · Grouped-query attention partitions query heads into groups whose members share key and value projections, reducing attention computation and memory use.
• Rotary Position Embeddingused by · Rotary Position Embedding is a positional encoding method that represents token positions by applying position-dependent rotations to components of attention query and key vectors.
• inner productuses · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
• key-value cachepart of · A key-value cache is a data structure that stores previously computed transformer attention keys and values for reuse during subsequent token generation.
• queryused by · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.