Attention helps a model identify which elements of an input sequence matter most for the current computation. In a Transformer, each word is compared with other words so stronger relationships provide more useful context. section 1.4
A Transformer uses attention as part of a broader sequence-processing workflow:
- Converts words into numerical representations. section 1.4
- Tracks word positions so sequence order contributes to meaning. section 1.4
- Checks relationships among words and emphasizes closely related elements. section 1.4
- Processes entire sequences in parallel through multiple self-attention layers. section 1.4
Context in a sentence
In “The dog chased the cat,” attention helps a Transformer relate “dog,” “chased,” and “cat” while interpreting the sentence’s meaning. section 1.4
Figure 1.2 A transformer takes a sentence and analyzes word positions and relationships to try and extract meaning from the text it sees.
Use the figure to trace how a Transformer analyzes word positions and relationships before producing meaning from a sentence. section 1.4
