Masked language modeling (MLM) is a self-supervised technique: a model predicts tokens hidden in an input sequence, using the original sequence as its training signal. For proteins, the tokens represent amino acids, and no external labels are needed. section 12.3.3
Predicting a missing amino acid
Given the protein sequence MKTAYIAKQRQISF, hide the K to produce MKTAYIA[MASK]QRQISF. The model uses the surrounding sequence to predict the missing amino acid. In one prediction, it assigned K a probability of 0.721, followed by R at 0.178 and Q at 0.064. section 12.3.3
Protein sequences are not random strings: evolution favors patterns compatible with stable, functional structures. Predicting missing residues encourages a model to learn recurring patterns, residues that appear together, and relationships between distant positions. The resulting representations can support downstream tasks, such as classifying antibodies; MLM alone does not make that classification decision. section 12.3.3 section 12.4
A small protein MLM follows this workflow:
- Start with unlabeled protein sequences and hide selected amino acids. Their original identities provide the targets for prediction. section 12.3.3
- Convert tokens into embeddings, add positional information, and use a transformer encoder's self-attention to place each residue in context. A prediction head produces a distribution over amino acids at each position. section 12.3.3
- Evaluate masked-token accuracy or pseudo-perplexity on held-out sequences. Check for near-duplicate sequences across the split, which can leak information into evaluation. section 12.3.3
Listing 12.1 Protein Language Model using Transformer Encoder
class ProteinLanguageModel(nn.Module):
def __init__(
self, vocab_size,
padding_idx, embedding_dim,
num_layers, num_heads,
ffn_dim, dropout,
):
super().__init__()
self.embedding = nn.Embedding( #A
vocab_size, embedding_dim, padding_idx #A
) #A
self.pe = PositionalEncoding(embedding_dim) #B
self.encoder = Encoder( #C
num_layers=num_layers, #C
embedding_dim=embedding_dim, #C
num_heads=num_heads, #C
ffn_dim=ffn_dim, #C
dropout=dropout, #C
) #C
self.lm_head = nn.Linear( #D
embedding_dim, vocab_size #D
) #D
def forward(self, batch):
input_ids = batch["input_ids"]
attention_mask = batch["attention_mask"]
x = self.embedding(input_ids) #E
x = self.pe(x) #F
x = self.encoder(x, attention_mask) #G
logits = self.lm_head(x) #H
return logitsInspect the PyTorch model components: token embeddings, positional encoding, a transformer encoder, and a prediction head. section 12.3.3
The same idea scales to ESM-2, an encoder-only transformer family trained on roughly 65 million clustered protein sequences. Its learned representations can support applications such as variant-effect prediction and secondary-structure prediction. Training data biases remain a limitation, so useful predictions still require task-appropriate evaluation. section 12.3.5 section 13.2.1 section 12.3.3





