Attention is a mechanism for weighting which parts of a sequence matter when processing a particular part or producing an output. In Transformers, self-attention relates positions within a sequence so each token can use contextual relationships rather than relying only on nearby or previously processed tokens. This capacity to consider long-range dependencies is central to contextual language understanding. section 2.0 section 2.5
Unlike recurrent networks, which process words one at a time while carrying a hidden state forward, Transformers process a sequence in parallel and use self-attention to identify relevant relationships across it. Parallel processing improves training and inference efficiency, while direct relationships across distant positions help preserve context that sequential models may lose. section 2.1
Figure 2.1 The strength of the transformer model is in its ability to understand the significance and context of every word in a sentence, unlike RNNs which primarily assess the connection of a word with its immediate neighbor.
Compare parallel contextual attention with the more local, sequential relationship handling associated with RNNs. section 2.1
Self-attention sits inside the Transformer’s encoder and decoder architecture, alongside input embeddings and positional encoding. Input embeddings turn tokens into high-dimensional numerical representations; attention then lets positions use one another’s representations. Multi-head attention supplies a broader view of these relationships, while positional encoding preserves information about token order. section 2.2 section 2.5
Attention takes different roles across Transformer components:
- Encoder self-attention builds contextual representations from an input sequence and supports semantic and syntactic feature extraction. section 2.2 section 2.5
- Decoder self-attention connects generated tokens, but masks future positions so next-token prediction can use only earlier outputs. section 2.3.2
- Encoder-decoder attention lets a decoder focus on relevant encoded input while producing an output sequence. section 2.3.2
Reading a sentence as a whole
For “The quick brown fox jumps over the lazy dog,” a recurrent model advances word by word. A Transformer instead examines the full sentence in parallel, using self-attention to focus on words and relationships relevant to each position. This is especially useful when an early word affects the meaning of a much later one. section 2.1
Attention also creates a systems constraint for embedding models. With a fixed context window, self-attention must compute relationships between every token pair, giving quadratic growth in attention work as sequence length grows. At 8,192 tokens, the cited embedding example requires roughly 67 million attention computations; doubling length to 16,384 raises this to about 268 million. Longer inputs also force more information into the same fixed-size embedding vector. section 6.3.1 section 6.3.2
Figure 6.3 Computational Cost of Context Window Expansion
Use the cost comparison to assess why expanding an embedding model’s context window rapidly increases compute requirements. section 6.3.2







































