Evolutionary Scale Modeling (ESM) is a family of protein language models trained on large collections of amino acid sequences. Like words in a sentence, amino acids form sequences whose patterns can reveal biological properties. section 12.3.5
ESM learns from unlabeled sequences by predicting missing amino acids. Scaling this self-supervised task across diverse proteins helps the model learn plausible sequence combinations, local structural patterns, and longer-range dependencies relevant to function. Its representations can support tasks such as predicting structure, stability, binding affinity, and mutation effects. section 12.3.5
Training at this scale requires substantial computation. A practical alternative is to load a pretrained ESM model, compute representations for protein sequences, and fine-tune it for a specific task. section 12.3.5
Listing 12.5 Loading ESM-2 and Computing Protein Embeddings
from transformers import AutoTokenizer, AutoModel
model_name = "facebook/esm2_t33_650M_UR50D"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name) #A
model.eval()
sequence = "MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQ"
inputs = tokenizer(sequence, return_tensors="pt") #B
with torch.no_grad():
outputs = model(**inputs) #C
embeddings = outputs.last_hidden_state #DLoad ESM-2 with Transformers, tokenize a protein sequence, and extract its hidden-state embeddings. The listing also requires importing torch. section 12.3.5
Figure 12.12 A zero-shot mutation landscape predicted by ESM-2. The x-axis represents the residue position, and the y-axis represents the 20 possible AA mutations. The color indicates the Log-Likelihood Ratio (LLR): brown regions indicate deleterious mutations that the model views as evolutionarily unlikely, while white/blue regions indicate neutral or beneficial mutations.

A zero-shot mutation landscape uses ESM-2 likelihood ratios to visualize how plausible substitutions are at each residue position. These are model predictions, not experimental validation. section 12.3.5
