A transformer is a neural-network architecture that models relationships among sequence elements using attention and position-wise transformations. In protein modeling, its input can be an amino acid sequence: a chain of chemical building blocks represented by letters. The output can be a contextual representation, or embedding, for each position. section 12.1.1 section 12.3.2
Self-attention lets each position gather information from other positions in the sequence. This is useful for proteins because amino acids far apart in the sequence can become neighbors in the folded structure. Unlike a recurrent model, a transformer does not process the sequence through recurrence; an encoder stack can process positions in parallel. section 12.3 section 12.3.2
Two common arrangements serve different sequence tasks:
- An encoder–decoder transformer uses an encoder to represent the input and a decoder to generate an output sequence one token at a time. Cross-attention gives the decoder access to the encoder’s token representations. section 12.3.2
- An encoder-only transformer omits the decoder and produces embeddings for the input sequence. Protein language models use these representations for tasks such as folding and property prediction. section 12.3.2
Figure 12.9 The transformer architecture. While the original transformer includes both an encoder and decoder for translation tasks, protein language modeling primarily uses the encoder-only architecture (left). This stack processes the entire sequence in parallel to generate embeddings for every residue.

Compare the encoder–decoder arrangement with the encoder-only stack used for protein representations. section 12.3.2
Attention uses learned queries, keys, and values to combine information across positions. Multiple attention heads provide several such combinations, while positional encoding adds position information to the input embeddings. section 12.3.2
From protein sequence to predicted structure
SimpleFold takes an amino acid sequence and uses pretrained ESM-2 embeddings as input to a folding pipeline. A generative method called flow matching guides noise toward three-dimensional atomic coordinates, with a confidence estimate for each residue. Transformers are only one ingredient: evaluate structures on held-out targets and treat low-confidence regions cautiously. section 12.2
Protein language models can learn useful sequence representations by masking amino acids and predicting the hidden entries. Those representations can then be adapted to tasks such as antibody classification, including distinguishing antibodies targeting HIV-1 from those targeting SARS-CoV-2. section 12.4







































