Attention weight

As explained in Build a Large Language Model (From Scratch)

A normalized relevance coefficient derived from attention scores that determines how much each value contributes to the context vector.

An attention weight is a normalized relevance coefficient that determines how strongly one input contributes to a context vector. Dot products produce attention scores; normalization converts those scores into positive values that sum to 1, making them interpretable as relative importance. section 3.3.1 section 3.7

Figure 3.7 The goal of self-attention is to compute a context vector for each input element that combines information from all other input elements. In this example, we compute the context vector z⁽²⁾. The importance or contribution of each input element for computing z⁽²⁾ is determined by the attention weights a₂₁ to a_2T. When computing z⁽²⁾, the attention weights are calculated with respect to input element x⁽²⁾ and all other inputs.

Use the diagram to trace how the weights for query x(2) determine each input's contribution to context vector z(2). section 3.3.1

The attention-weight computation follows this data flow:

• Compute an attention score for each input relative to a query, using a dot product as a similarity measure. section 3.3.1
• Normalize the scores with softmax so each row sums to 1 and every weight remains positive; scaled dot-product attention divides scores by the square root of the key dimension first. section 3.3.1 section 3.4.1
• Multiply the weights by the corresponding value vectors and sum them to produce the context vector. section 3.4.1
• For all tokens, form scores with matrix multiplication, apply softmax across the last dimension, and multiply the resulting weight matrix by the values. section 3.3.2

Six-token worked example

For query x(2) in “Your journey starts with one step,” the raw scores are [0.9544, 1.4950, 1.4754, 0.8434, 0.7070, 1.0865]. Softmax produces [0.1385, 0.2379, 0.2333, 0.1240, 0.1082, 0.1581], which sum to 1. Multiplying the six input vectors by these weights yields context vector [0.4419, 0.6515, 0.5683]. section 3.3.1 section 3.3.1

In trainable self-attention, the inputs are projected into query, key, and value vectors with Wq, Wk, and Wv. The attention weights are dynamic, context-specific coefficients, unlike the learned weight parameters in those matrices. section 3.4.1 section 3.4.1

How Build a Large Language Model (From Scratch) covers it

How deep it goes

Derives weights from dot products through softmax and matrix operations, then extends them to projected queries and keys.

Examples

Demonstrates one query over six token vectors, row-wise normalization, vectorized weights, and a resulting three-dimensional context vector.

What it assumes

Assumes vector inputs and dot products, while teaching score normalization, tensor dimensions, and query-key-value projections.

What you can do after

Build it from scratch with PyTorch using dot products, softmax, projected queries, keys, values, and matrix multiplication.

How it connects

Connects attention scores, softmax, query-key-value vectors, context vectors, causal attention masks, dropout, and multi-head attention.

liveBookmanning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A normalized attention score that determines how strongly a value contributes to an attention output.

Compare how 2 books explain attention weight →

Related concepts

• softmaxproduced by · Softmax is a function that exponentiates a vector of scores and normalizes the results into nonnegative values that sum to one.
• attention scoreproduced by · An attention score is a numerical compatibility or relevance value computed between elements, such as a query and key, before normalization into attention weights.
• valueuses · In attention, a value is a vector representation of input content that is combined with other values according to attention weights to produce an output.
• context vectorproduces · A context vector is a numerical representation that summarizes information from an input or surrounding context relevant to a model’s prediction or subsequent processing.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.