liveBook · Manning

Encoder-only transformer

As explained in Build AI Drug Discovery Pipelines

An encoder-only transformer is a transformer consisting of encoder layers that produces contextual representations without an autoregressive decoder.

An encoder-only transformer processes a sequence to produce a contextual representation for each token. For a protein, the tokens can be amino acid residues, and the outputs are embeddings: vectors representing those residues in context. Models such as ESM-2 use this approach for protein folding and property prediction. section 12.3.1 section 12.3.2

Unlike a recurrent neural network, a transformer does not process the sequence through recurrence. Its encoder stack can process the full sequence in parallel. Attention lets representations draw on information from across the sequence, rather than relying on a single final hidden state. section 12.3.2 section 12.3.1

A standard encoder-decoder transformer also has a decoder that generates a target sequence one token at a time, using cross-attention to access the encoder’s representations. An encoder-only transformer discards that decoder when the goal is to represent an existing sequence rather than generate a new one. section 12.3.2

Figure 12.9 The transformer architecture. While the original transformer includes both an encoder and decoder for translation tasks, protein language modeling primarily uses the encoder-only architecture (left). This stack processes the entire sequence in parallel to generate embeddings for every residue.

The diagram contrasts the encoder-only stack with the full encoder-decoder architecture and shows embeddings produced for each residue. section 12.3.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains parallel encoding and attention components, contrasting the encoder-only architecture with recurrent and encoder-decoder models.
Examples
Uses ESM-2 and protein sequences to illustrate residue-level representations.
What you can do after
Evaluate/understand when an encoder-only architecture is appropriate instead of an encoder-decoder model.
How it connects
Connects the encoder-only transformer to recurrent neural networks, cross-attention, and positional encoding.

Also explained in

Compare how 7 books explain encoder-only transformer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.