liveBook · Manning

Attention

As explained in Grokking AI Algorithms, Second Edition

Attention is a neural-network mechanism that weights relationships among input representations so relevant information contributes more strongly to a computation.

Attention lets a neural network weigh relationships among inputs so that relevant information contributes more strongly to a result. Think of highlighting important words while reading: the useful words can change depending on what you are trying to understand. section 11.1

In a Transformer, self-attention lets each token—a word or word fragment—draw information from other tokens in the sequence. For next-token prediction, the model can focus on relevant earlier tokens. Attention heads provide different perspectives on relationships, while stacked Transformer blocks can refine representations further. section 11.11 section 11.1 section 11.28

Following a relationship

In “Alice was beginning to get very tired,” understanding “tired” benefits from connecting it to “Alice,” who is tired, and “very,” which describes how tired. Self-attention gives tokens a way to weight such relationships when forming their representations. section 11.17

A single scaled dot-product attention head works in small steps:

  • Project each token’s embedding—a vector representing the token—into a query, key, and value using learned weight matrices. The query looks for relevant information, the key helps identify it, and the value carries the contribution. section 11.18
  • Compare each query with keys using dot products. Scale the scores by the square root of the vector dimension, then use softmax to turn them into attention weights. section 11.21
  • Multiply each value by its attention weight and add the results. The resulting context vector blends information for the querying token. section 11.21

A weighted contribution

In a four-value example, one token has an attention weight of 0.0632 and a value whose second component is 1.9940. Its contribution to that component is about 0.1261. Adding the corresponding contributions from all tokens produces the context vector. section 11.21

Self-attention alone does not tell the model token order. Positional encoding adds a position signal to each embedding, helping distinguish sentences such as “The dog bit the man” and “The man bit the dog.” section 11.15

Attention also connects language to images. In a U-Net, cross-attention can use image features as queries and a text embedding as keys and values. As the image is refined, its regions can draw on relevant parts of a prompt—for example, the word “snail” for a rounded patch. section 12.29

Figure 12.56 How cross-attention fits into the U-Net architecture
12_56

See where cross-attention fits between the U-Net’s downsampling and upsampling layers. section 12.29

How Grokking AI Algorithms, Second Edition covers it

How deep it goes
Works through single-head query, key and value projections, scaled scores, softmax weights, and context-vector calculations.
Examples
Traces relationships in a sentence about Alice, numerical token weights and context vectors, and a text prompt for a generated image.
What you can do after
Evaluate and understand how attention weights combine token information and how cross-attention links image features to text.
How it connects
Connects attention to Transformers, positional encoding, feed-forward networks, and U-Nets.

Also explained in

Compare how 29 books explain attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking AI Algorithms, Second Edition. The section links point to the parts of the book it draws on.