Relevance concerns what matters in a particular context, but it does not have one universal meaning or measure. Human learning is shaped by consequences and stakes; language quality includes staying on topic; and attention mechanisms use scores to compare tokens. section 5.4.1 section 2.6.6 section 4.1.3
In self-attention, a token’s query is compared with other tokens’ keys using a dot product. The resulting scores indicate relative alignment in the model’s internal representations, not whether a response is relevant to a person’s needs. A softmax converts each row of scores into attention weights, which determine how much information the corresponding value vectors contribute. section 4.1.3
“Rises” and “sun”
The query for “rises” may align more strongly with the key for “sun” than with “the,” giving “sun” a higher relevance score and, after softmax, a greater attention weight. section 4.1.3
















