liveBook · Manning

Attention weight

As explained in Timeless Algorithms, The Foundational Ideas

An attention weight is a normalized value indicating how strongly one element contributes to an attention-based representation.

When a model encounters a pronoun such as “it,” different tokens in the available context may contribute unequally to its representation. An attention weight is a normalized value that determines how much one token’s value contributes to that representation. section 12.2.1

Self-attention compares a query with candidate keys using dot products. These scores are divided by the square root of the key-vector dimension to prevent them from becoming excessively large. The resulting compatibility scores, or attention logits, are not yet normalized: they can be negative or positive and need not sum to one. section 12.2.1

softmax⁡(si)=esi∑j=1nesj

Softmax converts each score sᵢ into a weight by dividing its exponential by the sum of exponentials across the candidate tokens. The weights sum to one. section 12.2.1

zit=αtrophyvtrophy+αboxvbox

The contextual representation for “it” is a weighted sum of the candidate value vectors. A larger weight gives that value a larger contribution. section 12.2.1

Resolving a pronoun

In a simplified comparison, the query for “it” has dot products of 0.2 with the key for “trophy” and 0.8 with the key for “box.” With a key dimension of 2, the scaled scores are approximately 0.14 and 0.57. Softmax gives weights of 0.39 and 0.61. Using value vectors [1, 0] and [0, 1], respectively, the weighted sum is [0.39, 0.61]: both candidates contribute, but “box” contributes more. section 12.2.1

Direct comparison can make distant relationships available without passing information through a long chain of recurrent updates, but it does not guarantee a correct interpretation. An attention weight indicates influence within a calculation, not a complete causal explanation of the final output. Relevant information can also receive insufficient attention or be interpreted incorrectly. section 12.2.1 section 12.4.2

A single calculation is also not the whole model. Trained transformers combine many attention heads with feed-forward networks and stacked layers; simplified vectors make the mechanism visible without reproducing that full behavior. section 12.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Works through scaled dot products, softmax normalization, and a weighted sum, while distinguishing the simplified calculation from a complete transformer.
Examples
Calculates weights for “trophy” and “box” in a pronoun-resolution example using simplified vectors.
What it assumes
Builds on an earlier introduction to logits and the sigmoid function.
What you can do after
Evaluate and understand how weights shape a contextual representation without treating them as a complete causal explanation of an output.
How it connects
Connects attention weights to self-attention, softmax, logits, recurrence, and contextual representations.

Also explained in

Compare how 7 books explain attention weight →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.