Self-supervised learning trains a model with prediction targets derived from the input data itself, rather than external labels. In drug discovery, it can turn unlabeled protein sequences or cellular images into useful representations: numerical features that capture patterns relevant to later tasks. section 12.3.3 section 13.2.4
For a protein language model, masked language modeling hides or alters selected amino acids and asks the model to predict the originals from the sequence context. This encourages it to learn recurring amino-acid patterns and dependencies between distant positions. The resulting representations can support tasks such as predicting protein structure, stability, binding affinity, or mutational effects. section 12.3.3 section 12.3.5
Predicting a missing amino acid
Given the protein sequence MKTAYIAKQRQISF, replace K with [MASK] to produce MKTAYIA[MASK]QRQISF. The model uses the surrounding sequence to predict K, the original amino acid. The sequence supplies the target; no separate annotation is needed. section 12.3.3
A masking and evaluation workflow:
- Randomly select 15% of sequence positions. Of those selected, replace 80% with [MASK], replace 10% with random amino acids, and leave 10% unchanged. section 12.3.3
- Train the model to predict the original amino acid at every selected position. The mixture reduces reliance on a mask token that will be absent when extracting embeddings from unmasked sequences. section 12.3.3
- Evaluate with masked-token accuracy or pseudo-perplexity on held-out sequences. Check for near-duplicates across splits, and remember that the model inherits biases in its training set. section 12.3.3
The same principle applies beyond sequences. For cellular images, a model can learn by predicting masked image regions or matching images of the same perturbation across different wells. These objectives can teach features related to cell shape, organelle organization, and biological mechanisms without requiring an annotation for every image. section 13.2.4
Listing 12.5 Loading ESM-2 and Computing Protein Embeddings
from transformers import AutoTokenizer, AutoModel
model_name = "facebook/esm2_t33_650M_UR50D"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name) #A
model.eval()
sequence = "MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQ"
inputs = tokenizer(sequence, return_tensors="pt") #B
with torch.no_grad():
outputs = model(**inputs) #C
embeddings = outputs.last_hidden_state #DLoad a pre-trained ESM-2 model and extract embeddings from a protein sequence, avoiding the cost of training a large model from scratch. section 12.3.5









