Attention is a neural-network mechanism that weighs the importance of input elements when producing an output. In Transformer models, self-attention considers the full input context simultaneously, helping the model connect words or other elements that are far apart. This supports language understanding while enabling parallel computation during training. section 2.3
Attention supports several practical capabilities, with important deployment trade-offs:
- Captures dependencies across distant words, supporting machine translation, summarization, and text generation. section 2.3
- Processes image regions through Vision Transformers, which apply self-attention globally across sequences of image patches. section 4.1.3
- Raises computational cost as input grows: global attention scales quadratically with token count or context length. section 4.1.3 section 2.8.6
- Favors the smallest context window that satisfies the task, because larger windows can increase memory use, computation, and latency. section 2.8.6
Predicting the next word
Given “The cat sat on the...,” a Transformer can assign substantial attention to “cat” when predicting “mat.” The example illustrates how different parts of the input receive different importance during generation. section 2.3
Figure 4.6 Vision transformer (ViT) architecture [2]
Use the ViT architecture figure to trace image patches and positional embeddings through linear projection, multi-head attention, normalization, and the Transformer encoder. section 4.1.3
