liveBook · Manning

Self-attention

As explained in Timeless Algorithms, The Foundational Ideas

Self-attention forms representations of sequence elements by relating each element to others in the same sequence.

A pronoun can depend on a noun several words away. Self-attention constructs a representation of each token by weighting information from other tokens in the same sequence, so its representation changes with context. Unlike a recurrent hidden state, these relationships need not pass through every intervening state update. section 12.1.2 section 12.2

Each token’s embedding is transformed by learned projections into a query, a key, and a value. The query expresses what the token seeks; keys signal relevance; values carry information to be combined. For a query q and key kᵢ, a scaled dot product gives a compatibility score sᵢ = (q · kᵢ) / √d, where d is the vector dimension. Softmax turns the scores into weights αᵢ = exp(sᵢ) / Σⱼ exp(sⱼ). The contextual representation is z = Σᵢ αᵢvᵢ: a weighted sum of the values. section 12.2.1

Resolving a pronoun

In a simplified example, the query for “it” is [0.2, 0.8], while the keys and values for “trophy” and “box” are [1, 0] and [0, 1]. Their dot products with the query are 0.2 and 0.8. After scaling and softmax, the weights are approximately 0.39 and 0.61, producing [0.39, 0.61]. The representation draws more strongly on “box.” These two-dimensional vectors and identity projections make the calculation transparent; they do not reproduce a full trained model. section 12.2.1 section 12.2

Self-attention is the central operation of the transformer, but it works alongside other components. Positional information distinguishes order, multiple heads can capture different relationships, and feed-forward networks further transform representations. In a decoder, causal masking prevents self-attention from accessing future tokens. section 12.1.2 section 12.1.2

Figure 12.1 The transformer architecture introduced in Attention Is All You Need. Stacked encoder and decoder layers combine multi-head attention with feed-forward transformations, residual connections, and normalization. Positional encodings preserve sequence order without recurrence. The decoder uses masked self-attention to prevent access to future tokens and cross-attention to incorporate encoder outputs. Within each multi-head attention block, individual heads use dot products to calculate compatibility scores and softmax to convert those scores into attention weights, as demonstrated in the worked examples that follow. Reproduced from Vaswani et al (2017), Attention Is All You Need, Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS 2017).

The architecture diagram locates self-attention within encoder and decoder layers and shows how masking and cross-attention differ. section 12.1.2

Attention operations can be computed largely in parallel during training, unlike recurrent state propagation. This supports scale, but context length, model capacity, training data, and computation still constrain what a model can use. Autoregressive language models generate tokens one at a time, and factual recall depends on learned components throughout the network—not on one attention calculation alone. section 10.5.2 section 12.2.2 section 12.6

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Works through projections, scaled dot products, softmax, and weighted values, while distinguishing simplified calculations from full transformer behavior.
Examples
Uses pronoun resolution, factual recall, and autoregressive generation to illustrate distinct contributions of attention.
What you can do after
Evaluate how attention weights produce contextual representations and why a single attention calculation cannot explain complete model behavior.
How it connects
Connects self-attention to recurrent neural networks, positional encodings, multi-head attention, feed-forward networks, and causal masking.

Also explained in

Compare how 5 books explain self-attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.