Attention score

As explained in Build a Large Language Model (From Scratch)

The numeric values computed to quantify how strongly one token should attend to another in attention mechanisms.

An attention score is an unnormalized compatibility value between a query token and an input token. In self-attention, dot products make larger scores indicate stronger potential contributions to the query’s context vector; softmax later converts each row of scores into attention weights. section 3.3.1 section 3.4.1

The computation follows a consistent tensor flow:

• With untrained self-attention, compute every score as the dot product between one input embedding and the query embedding; for query x(2), six inputs produce a length-six score vector. section 3.3.1
• For all tokens, assemble the pairwise scores as a matrix: inputs @ inputs.T. Each row corresponds to one query, and each column corresponds to a candidate input. section 3.3.2
• With trainable projections, transform inputs into queries and keys, then compute scores as queries @ keys.T rather than directly comparing input embeddings. section 3.4.1
• Scale scores by the square root of the key dimension before applying softmax; scaling limits overly sharp softmax outputs and helps avoid very small gradients. section 3.4.1
• Normalize each row with softmax to obtain attention weights, then use those weights to combine value vectors into context vectors. section 3.3.2 section 3.4.1

Six-token score vector

Using the second embedding as the query produces the unnormalized scores [0.9544, 1.4950, 1.4754, 0.8434, 0.7070, 1.0865]. Applying softmax yields [0.1385, 0.2379, 0.2333, 0.1240, 0.1082, 0.1581], whose values sum to 1. The corresponding weighted inputs produce the context vector [0.4419, 0.6515, 0.5683]. section 3.3.1

Figure 3.8 The overall goal is to illustrate the computation of the context vector z⁽²⁾ using the second input element, x⁽²⁾ as a query. This figure shows the first intermediate step, computing the attention scores w between the query x⁽²⁾ and all other input elements as a dot product. (Note that the numbers are truncated to one digit after the decimal point to reduce visual clutter.)

Visualizes attention scores as dot products between the second input query and every input element before normalization. section 3.3.1

How Build a Large Language Model (From Scratch) covers it

How deep it goes

Explains pairwise dot products, matrix shapes, scaling, and softmax, then stops short of detailed optimization internals.

Examples

Works through six three-dimensional token embeddings and computes one score vector, normalized row, and context vector.

What it assumes

Assumes token embeddings and basic vector or matrix multiplication, while deriving attention computation from simple examples.

What you can do after

Builds scaled dot-product attention in PyTorch using matrix multiplication, softmax, and projected queries, keys, and values.

How it connects

Connects attention scores with queries, keys, softmax, attention weights, value vectors, and context vectors.

liveBookmanning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Related concepts

• attention weightproduces · An attention weight is a computed coefficient, often normalized, that determines how strongly an input element or value contributes to an attention-based representation.
• matrix-matrix multiplicationused by · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.
• inner productproduced by · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.