Attention is a transformer mechanism that focuses computation on the most relevant earlier tokens. It behaves like a fuzzy dictionary: a query compares with many keys, weights their similarity, and combines the corresponding values. section 3.2.2
When generating the next token, attention works roughly like this:
- Treat the current token as a query and previous tokens as keys. section 3.2.2
- Compare the query with every key and assign greater weight to better matches. section 3.2.2
- Combine the values according to those weights to produce an output used for token prediction. section 3.2.2
Dictionary lookup, made fuzzy
A Python dictionary needs an exact key or returns nothing. Attention instead returns a weighted result from the most similar keys, so every query receives an approximate response.
