Context vector

As explained in Build a Large Language Model (From Scratch)

A vector representation formed by combining input information with attention weights to summarize relevant context for a token.

A context vector is an enriched embedding for one token. It combines that token’s information with information from other input tokens, using attention weights to determine each contribution. In self-attention, the result summarizes the context most relevant to the token and can be computed for every position in a sequence. section 3.3.1 section 3.7

Construct a context vector through this data flow:

• Represent the input sequence as an embedding matrix X, with one vector per token. section 3.3.1
• Select a query token and compare it with input elements using dot products to obtain attention scores. section 3.3.1
• Normalize the scores with softmax so the attention weights in each row sum to 1. section 3.3.2
• Compute the context vector as the weighted sum of input or value vectors. section 3.3.1 section 3.4.1
• Scale the mechanism to all tokens with matrix multiplication: attention weights shaped [T, T] multiplied by inputs or values shaped [T, d]. section 3.3.2

From “journey” to a context vector

For the six-token sequence “Your journey starts with one step,” choose x(2), the embedding for “journey,” as the query. The simplified mechanism produces attention weights [0.1385, 0.2379, 0.2333, 0.1240, 0.1082, 0.1581], which sum to 1. Multiplying these weights by the six input embeddings gives z(2) = [0.4419, 0.6515, 0.5683]. With trainable query, key, and value projections, the same process uses Q, K, and V and produces a two-dimensional context vector [0.3061, 0.8210]. section 3.3.1 section 3.4.1

Trainable self-attention projects each input into query, key, and value vectors with Wq, Wk, and Wv. It computes attention scores from queries and keys, applies scaled softmax, and combines the values. A PyTorch module returns one context vector per token; causal attention adds a mask so left-to-right language models cannot use future tokens. section 3.4.2 section 3.5.3

Figure 3.7 The goal of self-attention is to compute a context vector for each input element that combines information from all other input elements. In this example, we compute the context vector z⁽²⁾. The importance or contribution of each input element for computing z⁽²⁾ is determined by the attention weights a₂₁ to a_2T. When computing z⁽²⁾, the attention weights are calculated with respect to input element x⁽²⁾ and all other inputs.

Use the figure to trace how attention weights for query x(2) combine all input elements into context vector z(2). section 3.3.1

How Build a Large Language Model (From Scratch) covers it

How deep it goes

Explains dot products, softmax, matrix multiplication, trainable projections, and causal masking, then defers multi-head implementation.

Examples

Demonstrates “Your journey starts with one step,” six three-dimensional embeddings, explicit weights, tensor shapes, and PyTorch self-attention classes.

What it assumes

Assumes token embeddings and basic tensor operations, while introducing attention scores, weights, queries, keys, and values incrementally.

What you can do after

Build it from scratch using PyTorch dot products, softmax, matrix multiplication, trainable projections, and causal masking.

How it connects

Connects context vectors to self-attention, attention weights, queries, keys, values, softmax, causal attention, and multi-head attention.

liveBookmanning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Related concepts

• attention weightproduced by · An attention weight is a computed coefficient, often normalized, that determines how strongly an input element or value contributes to an attention-based representation.
• matrix-matrix multiplicationused by · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.
• parameterproduced by · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
• attentionproduced by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.