An encoder-only transformer processes an input sequence into rich, context-aware representations. Because each token can attend to the full context on both sides, it suits representation and classification tasks rather than autoregressive generation. section 3.1.1
Key characteristics include:
- Uses an encoder stack built from self-attention and feed-forward blocks. section 3.1.1
- Typically pairs with masked language modeling during pretraining and task-specific fine-tuning afterward. section 3.1.1
- Retains quadratic full-attention cost as sequence length grows, making very long contexts a practical bottleneck. section 3.1.1
Figure 3.2 Transformer model architecture. It consists of an encoder stack (left) and a decoder stack (right). The encoder processes the input sequence using multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks with residual connections and layer normalization (Add & LN). The decoder processes the target sequence using masked multi-head self-attention (MMSA) and multi-head cross attention (MCA) to integrate encoder features, then projects the output to probabilities through a linear layer and softmax. Source: [9]

Use the architecture figure to compare the encoder stack with decoder components, including self-attention, residual connections, and normalization. section 3.1.1






