Attention assigns relative importance to tokens, then uses those scores to let relevant context influence a token’s embedding. It helps disambiguate meaning within the context window. section 1.20
Disambiguating “bark”
In “dog’s bark,” attention focuses mainly on “dog,” then “bark,” then “’s.” Information from “dog” consequently rubs off on “bark,” clarifying its meaning. section 1.20
The mechanism works in three broad moves:
- Calculate an attention score for each token in the context. section 1.20
- Compare embeddings through mathematical operations such as projections. section 1.20
- Use the scores to guide how token information changes contextualized embeddings. section 1.20
Figure 1.10 The attention mechanism calculates the relative relevance of all tokens in the context window to contextualize or disambiguate the last token.
Use the figure to see how tokens receive relative relevance when contextualizing the final token. section 1.20
