An encoder-only transformer processes a sequence to produce a contextual representation for each token. For a protein, the tokens can be amino acid residues, and the outputs are embeddings: vectors representing those residues in context. Models such as ESM-2 use this approach for protein folding and property prediction. section 12.3.1 section 12.3.2
Unlike a recurrent neural network, a transformer does not process the sequence through recurrence. Its encoder stack can process the full sequence in parallel. Attention lets representations draw on information from across the sequence, rather than relying on a single final hidden state. section 12.3.2 section 12.3.1
A standard encoder-decoder transformer also has a decoder that generates a target sequence one token at a time, using cross-attention to access the encoder’s representations. An encoder-only transformer discards that decoder when the goal is to represent an existing sequence rather than generate a new one. section 12.3.2
Figure 12.9 The transformer architecture. While the original transformer includes both an encoder and decoder for translation tasks, protein language modeling primarily uses the encoder-only architecture (left). This stack processes the entire sequence in parallel to generate embeddings for every residue.

The diagram contrasts the encoder-only stack with the full encoder-decoder architecture and shows embeddings produced for each residue. section 12.3.2






