When translating “That cat chased a mouse,” a person focuses on “cat” and “mouse” rather than small connecting words. Attention applies a similar idea to neural networks: it assigns greater weight to input elements that matter most for producing a representation or output. section 1.2
In sequence-to-sequence translation, attention lets the decoder search across the entire input instead of processing it only piece by piece. Each position receives context from the sequence, helping the system connect words that may be far apart. section 1.2
Attention became especially important because it supports:
- Long-term dependencies, by incorporating the whole input context at each step. section 1.2
- Parallel computation, because transformer models replace lengthy sequential processing with attention information. section 1.2
- Self-attention, which repeatedly builds new representations using relationships across the whole input. section 1.2
Resolving context
The word “it” can receive different attention distributions depending on the surrounding words, illustrating how context changes which input elements matter.
Figure 1.2 The distribution of attention for the word “it” in different contexts.
Explore how attention assigned to “it” changes across contexts. section 1.2
