When a sentence contains a pronoun, its interpretation may depend on a noun much earlier in the sequence. Attention helps construct context by weighting relationships among elements, so relevant information can influence a representation directly rather than only through a carried-forward state. section 12.0 section 13.4.2
In an earlier sequence-to-sequence model, a decoder could use attention to assign different weights to encoder states instead of relying entirely on a single final state. Schematically, a context representation can be written as c = Σᵢ wᵢhᵢ, where hᵢ is an encoder state and wᵢ is its attention weight. The weights express how much each state contributes to the context. section 12.3.3
The transformer, introduced in 2017, did not invent attention. Its architectural shift was to remove recurrence and make self-attention the primary operation for constructing context. Tokens could relate directly to other tokens, rather than depending entirely on information passed through intermediate states. This organization also made parallel computation more practical during training. section 12.3.3 section 12.0
Direct relationships can connect information distributed across an input:
- In translation, a word or phrase can be aligned with relevant material elsewhere in a sentence. section 12.3.3
- In document analysis, claims can be related to qualifications and references in other passages. section 12.3.3
- In programming, a variable or function call can be related to an earlier definition. section 12.3.3
Self-attention alone did not produce modern language-model capabilities; larger datasets, deeper networks, and more computation also mattered. Attention can be costly when relationships are calculated across all tokens. More efficient variants limit, approximate, or selectively calculate relationships, while hybrid architectures combine attention with mechanisms such as memory or retrieval. section 12.3.3 section 12.4.2







































