For each token, attention generates a Query (Q), Key (K), and Value (V). Q searches for relevant information; K represents what a token offers; V carries its semantic content. section 3.1.2
Attention uses these vectors to enrich a token’s embedding:
- Compare the current token’s Query with each token’s Key using dot products to calculate compatibility scores. section 3.1.2
“Spring” searches for context
In “Fresh water flows from the mountain spring daily,” the Query for “spring” compares with the Keys of other tokens; the resulting scores determine which Values contribute to its context. section 3.1.2
