Attention, specifically self-attention, lets each token incorporate information from other tokens when forming a context-aware representation. It captures long-range dependencies without the sequential decay associated with recurrent networks and supports generation, representation learning, and sequence-to-sequence tasks. section 3.1.1
Self-attention computes a token representation in three practical stages:
- Compare queries with keys to produce compatibility scores. section 3.1.1
- Normalize the scores with softmax to obtain attention weights. section 3.1.1
- Combine values through a weighted sum to produce a context-aware output. section 3.1.1
Within a Transformer block, self-attention operates alongside token embeddings, positional information, feed-forward networks, residual connections, and normalization. This architecture forms representations that can later be specialized through fine-tuning. section 3.1.1 section 3.8
Context-aware token output
For a token sequence, the model compares each token's query with other tokens' keys, converts those comparisons into weights, and blends the corresponding values. Tokens that receive larger weights contribute more strongly to the resulting representation. section 3.1.1
Figure 3.1 Encoder-decoder RNN architecture for language translation. The encoder processes the input sequence sequentially, compressing all information into a single hidden state. The decoder then uses this bottleneck representation to generate the output. A key limitation is that the decoder cannot directly access earlier encoder states. It relies entirely on the compressed final hidden state, which can lead to information loss in longer sequences. Image from Sebastian Raschka’s book Build a Large Language Model (From Scratch) [8].

Use the recurrent encoder-decoder contrast to see why attention helps: recurrent encoding compresses a sequence into one final state, creating an information bottleneck for longer inputs. section 3.1.1







































