liveBook · Manning

ESM

As explained in Build AI Drug Discovery Pipelines

Evolutionary Scale Modeling is a family of protein language models trained on large collections of amino acid sequences.

Evolutionary Scale Modeling (ESM) is a family of protein language models trained on large collections of amino acid sequences. Like words in a sentence, amino acids form sequences whose patterns can reveal biological properties. section 12.3.5

ESM learns from unlabeled sequences by predicting missing amino acids. Scaling this self-supervised task across diverse proteins helps the model learn plausible sequence combinations, local structural patterns, and longer-range dependencies relevant to function. Its representations can support tasks such as predicting structure, stability, binding affinity, and mutation effects. section 12.3.5

Training at this scale requires substantial computation. A practical alternative is to load a pretrained ESM model, compute representations for protein sequences, and fine-tune it for a specific task. section 12.3.5

Listing 12.5 Loading ESM-2 and Computing Protein Embeddings
from transformers import AutoTokenizer, AutoModel
 
model_name = "facebook/esm2_t33_650M_UR50D"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)    #A
model.eval()
 
sequence = "MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQ"
inputs = tokenizer(sequence, return_tensors="pt")    #B
with torch.no_grad():
    outputs = model(**inputs)    #C
embeddings = outputs.last_hidden_state    #D

Load ESM-2 with Transformers, tokenize a protein sequence, and extract its hidden-state embeddings. The listing also requires importing torch. section 12.3.5

Figure 12.12 A zero-shot mutation landscape predicted by ESM-2. The x-axis represents the residue position, and the y-axis represents the 20 possible AA mutations. The color indicates the Log-Likelihood Ratio (LLR): brown regions indicate deleterious mutations that the model views as evolutionarily unlikely, while white/blue regions indicate neutral or beneficial mutations.

A zero-shot mutation landscape uses ESM-2 likelihood ratios to visualize how plausible substitutions are at each residue position. These are model predictions, not experimental validation. section 12.3.5

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains masked amino acid prediction and demonstrates pretrained embedding extraction and a classification-head setup.
Examples
Demonstrates ESM-2 embeddings for a protein sequence and visualizes a zero-shot mutation landscape.
What you can do after
Apply pretrained ESM-2 via Transformers to extract protein embeddings for downstream tasks.
How it connects
Connects protein language models with self-supervised learning, masked language modeling, and fine-tuning.

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.