liveBook · Manning

Transformer architecture

As explained in Build AI Drug Discovery Pipelines

A transformer architecture is a neural-network architecture that models relationships among sequence elements using attention and position-wise transformations, potentially with encoder and decoder stacks.

A transformer is a neural-network architecture that models relationships among sequence elements using attention and position-wise transformations. In protein modeling, its input can be an amino acid sequence: a chain of chemical building blocks represented by letters. The output can be a contextual representation, or embedding, for each position. section 12.1.1 section 12.3.2

Self-attention lets each position gather information from other positions in the sequence. This is useful for proteins because amino acids far apart in the sequence can become neighbors in the folded structure. Unlike a recurrent model, a transformer does not process the sequence through recurrence; an encoder stack can process positions in parallel. section 12.3 section 12.3.2

Two common arrangements serve different sequence tasks:

  • An encoder–decoder transformer uses an encoder to represent the input and a decoder to generate an output sequence one token at a time. Cross-attention gives the decoder access to the encoder’s token representations. section 12.3.2
  • An encoder-only transformer omits the decoder and produces embeddings for the input sequence. Protein language models use these representations for tasks such as folding and property prediction. section 12.3.2
Figure 12.9 The transformer architecture. While the original transformer includes both an encoder and decoder for translation tasks, protein language modeling primarily uses the encoder-only architecture (left). This stack processes the entire sequence in parallel to generate embeddings for every residue.

Compare the encoder–decoder arrangement with the encoder-only stack used for protein representations. section 12.3.2

Attention uses learned queries, keys, and values to combine information across positions. Multiple attention heads provide several such combinations, while positional encoding adds position information to the input embeddings. section 12.3.2

From protein sequence to predicted structure

SimpleFold takes an amino acid sequence and uses pretrained ESM-2 embeddings as input to a folding pipeline. A generative method called flow matching guides noise toward three-dimensional atomic coordinates, with a confidence estimate for each residue. Transformers are only one ingredient: evaluate structures on held-out targets and treat low-confidence regions cautiously. section 12.2

Protein language models can learn useful sequence representations by masking amino acids and predicting the hidden entries. Those representations can then be adapted to tasks such as antibody classification, including distinguishing antibodies targeting HIV-1 from those targeting SARS-CoV-2. section 12.4

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains encoder–decoder and encoder-only architectures, attention, and positional encoding, while focusing on one component of the wider folding pipeline.
Examples
Uses SimpleFold and ESM-2 for protein folding and antibody classification to illustrate downstream use of protein language models.
What it assumes
Introduces amino acid sequences and protein structure before contrasting transformers with recurrent models and variational autoencoders.
What you can do after
Evaluate and understand how sequence embeddings support protein folding predictions, including the importance of held-out targets and per-residue confidence.
How it connects
Connects self-attention with recurrent neural networks, cross-attention, protein language models, and flow matching.

Also explained in

Compare how 40 books explain transformer architecture →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.