liveBook · Manning

Encoder-only transformer

As explained in Building Specialized AI Systems

An encoder-only model consists of encoder layers that can represent each token using context from both directions.

An encoder-only transformer processes an input sequence into rich, context-aware representations. Because each token can attend to the full context on both sides, it suits representation and classification tasks rather than autoregressive generation. section 3.1.1

Key characteristics include:

  • Uses an encoder stack built from self-attention and feed-forward blocks. section 3.1.1
  • Typically pairs with masked language modeling during pretraining and task-specific fine-tuning afterward. section 3.1.1
  • Retains quadratic full-attention cost as sequence length grows, making very long contexts a practical bottleneck. section 3.1.1
Figure 3.2 Transformer model architecture. It consists of an encoder stack (left) and a decoder stack (right). The encoder processes the input sequence using multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks with residual connections and layer normalization (Add & LN). The decoder processes the target sequence using masked multi-head self-attention (MMSA) and multi-head cross attention (MCA) to integrate encoder features, then projects the output to probabilities through a linear layer and softmax. Source: [9]

Use the architecture figure to compare the encoder stack with decoder components, including self-attention, residual connections, and normalization. section 3.1.1

How Building Specialized AI Systems covers it

How deep it goes
Explains bidirectional attention, pretraining objectives, architecture choice, and long-context cost, without detailing a specific implementation.
Examples
Illustrates the layout with an encoder–decoder Transformer diagram showing self-attention, feed-forward blocks, residual connections, and normalization.
What you can do after
Evaluate whether bidirectional representations fit classification and representation tasks, while recognizing long-context attention costs.
How it connects
Contrasts with decoder-only models and encoder–decoder architectures, and connects to self-attention, cross-attention, and masked language modeling.

Also explained in

Compare how 7 books explain encoder-only transformer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.