liveBook · Manning

Chapter 12: Transformers for Protein Structure Prediction

What you will learn in chapter 12 of Build AI Drug Discovery Pipelines

This introduction explains why protein structures matter to drug discovery and why the gap between known sequences and experimentally determined structures motivates computational prediction.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains why protein structures matter to drug discovery and why the gap between known sequences and experimentally determined structures motivates computational prediction.

Key terms: three-dimensional protein structures, drug-target binding affinity, structure determines function, experimental structure determination, Protein Data Bank (PDB), protein sequences, computationally predicted structures, experimental validation, SARS-CoV-2, spike protein

12.1 Conjoined Problems: Structure Prediction & Protein Design

This section frames protein structure prediction and protein design as inverse problems that share a vast conformational search challenge.

Key terms: protein structure prediction, protein design, amino acid sequence, three-dimensional structure, conformations, exhaustive search, functional structures

12.1.1 Protein Structure Essentials

This section explains how amino acid sequences form functional three-dimensional protein structures and what a structure-prediction model must capture.

Key terms: AA sequence, three-dimensional structure, active site, primary structure, secondary structure, tertiary structure, quaternary structure, peptide bonds, backbone, sidechain

12.1.2 Why Protein Folding is a Hard Modeling Problem

This section explains why predicting and designing protein structures are difficult, from vast search spaces and rugged energy landscapes to the limits of earlier methods and the appeal of learned sequence-to-structure patterns.

Key terms: amino acids, three-dimensional shape, local patterns, forward problem, protein structure prediction, inverse problem, protein design, phi and psi torsion angles, backbone conformations, Levinthal's paradox

12.2 SimpleFold: An End-to-End Example

This section traces how SimpleFold turns an amino acid sequence into a predicted protein structure and explains how to interpret the result.

Key terms: amino acid sequence, ESM-2, per-residue embedding, multiple sequence alignment (MSA), flow matching, velocity field, iterative refinement, all-atom coordinates, per-residue confidence estimate, evolutionary signal

12.3 Protein Language Models

This section explains why protein sequences need long-range attention and how transformer attention extends the graph attention used in Chapter 11.

Key terms: AAs, beta sheets, disulfide bonds, global optimization problem, graph attention networks (GATs), transformers, attention weights, weighted aggregation, global attention

12.3.1 Why Transformers? From Local to Global Attention

The section explains why compressing a long protein sequence into one recurrent hidden state loses information and how cross-attention lets a decoder draw on all encoder states.

Key terms: encoding bottleneck, hidden state, context vector, GRUs, local context, encoder hidden states, cross-attention, query (Q), keys (K), values (V)

12.3.2 The Encoder-Only Transformer Architecture

Explains how an encoder-only Transformer turns a protein sequence into contextual residue representations using self-attention, positional information, and stacked encoder blocks.

Key terms: encoder-only transformer, encoder, decoder, embeddings, self-attention, queries, keys, values, long-range dependencies, scaled dot-product attention

12.3.3 Training a Small Protein Language Model

This section explains how a small protein language model learns from masked amino acids and how to interpret its predictions and attention patterns.

Key terms: masked language modeling, self-supervised learning, amino acids, [MASK] token, masking strategy, unmasked sequences, embedding layer, positional encoding, transformer encoder, language modeling head

12.3.4 Downstream Applications: Antibody Classification

This section adapts a protein Transformer to classify antibody sequences by virus target, then interprets its predictions, performance, and limitations.

Key terms: antibody heavy chain, supervised classification, masked language modeling, virus target, language modeling head, mean pooling, attention mask, classification head, padding tokens, cross-entropy loss

12.3.5 The Evolutionary Scale Modeling (ESM) Family of Models

This section explains how scaled protein language models provide reusable protein representations for classification, mutation scoring, and structure prediction.

Key terms: ESM-2, encoder-only Transformer, masked language modeling, emergent structure, rotary position embeddings, per-residue embeddings, mean pooling, classification head, feature extraction, fine-tuning

12.4 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 11: Graph Neural Networks for Predicting Drug-Target Affinity Chapter 13: Multimodal AI Systems for End-to-End Drug Discovery →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.