liveBook · Manning

ESM-2

As explained in Build AI Drug Discovery Pipelines

ESM-2 is a protein language model trained on large collections of amino acid sequences to learn sequence representations and predict mutation effects.

ESM-2 is a family of protein language models that learns from amino acid sequences. Its representations can help predict mutation effects or supply structural information to a separate folding model—useful when a protein target lacks an experimental structure. section 12.3.5 section 13.3.2

ESM-2 uses an encoder-only Transformer trained by masked language modeling: it predicts an amino acid hidden in a sequence from its surrounding context. Training on roughly 65 million clustered UniRef50 sequences helps it learn representations associated with protein evolution, structure, and function. Released in 2022, its variants range from 8 million to 15 billion parameters and use rotary position embeddings in attention. section 12.3.5

An embedding is a numerical representation of a sequence or residue. To obtain one, load a pretrained ESM-2 model and tokenizer, tokenize an amino acid sequence, and run the model in evaluation mode without gradients. The last hidden states provide the embeddings; they are not themselves 3D coordinates. section 12.3.5 section 13.3.2

Listing 12.5 Loading ESM-2 and Computing Protein Embeddings
from transformers import AutoTokenizer, AutoModel
 
model_name = "facebook/esm2_t33_650M_UR50D"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)    #A
model.eval()
 
sequence = "MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQ"
inputs = tokenizer(sequence, return_tensors="pt")    #B
with torch.no_grad():
    outputs = model(**inputs)    #C
embeddings = outputs.last_hidden_state    #D

This Hugging Face Transformers listing loads the 650-million-parameter model and extracts hidden-state embeddings. Import PyTorch as torch before running it. section 12.3.5

For mutation scoring, mask a position and compare the model's predicted probabilities for a proposed amino acid and the original, or wild-type, amino acid. The log-likelihood ratio (LLR) is log P(mutant | context) minus log P(wild type | context). A negative value means the mutant is less plausible to the model; a positive value means it is more plausible. section 12.3.5 section 13.3.3

Scoring a substitution

If the model assigns probability 0.03 to a proposed amino acid and 0.62 to the wild type, the LLR is approximately −3.06. This favors the original residue in that sequence context; it is a computational indication, not an experimental measurement of the mutation's effect. section 12.3.5 section 12.4

Figure 12.12 A zero-shot mutation landscape predicted by ESM-2. The x-axis represents the residue position, and the y-axis represents the 20 possible AA mutations. The color indicates the Log-Likelihood Ratio (LLR): brown regions indicate deleterious mutations that the model views as evolutionarily unlikely, while white/blue regions indicate neutral or beneficial mutations.

A mutation landscape displays scores across residue positions and possible substitutions, helping identify constrained and variable regions. section 12.3.5 section 12.4

ESM-2 embeddings can also support downstream prediction. SimpleFold uses per-residue embeddings to help predict all-atom coordinates and confidence estimates without a multiple sequence alignment. In a virtual-screening workflow, ESM-2 embeddings of FADS1 fed into ESMFold, whose separately trained structure-prediction head produced coordinates for docking. Such structures are predictions to validate, not experimental structures. section 12.2 section 13.3.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains masked-language training and mutation likelihoods, and provides pretrained embedding extraction and an optional classification-head listing.
Examples
Illustrates a substitution likelihood calculation, a mutation landscape, and FADS1 structure generation for virtual screening.
What you can do after
Applies ESM-2 via Hugging Face Transformers to extract protein embeddings for downstream tasks.
How it connects
Connects masked language modeling and transfer learning with protein structure prediction and mutation effect prediction.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.