A protein language model (PLM) learns biological patterns from unlabeled amino-acid sequences. Masked language modeling hides residues and trains the model to predict them, producing contextual representations that capture sequence constraints, structural tendencies, and long-range dependencies. section 12.3.3 section 12.4
A practical PLM workflow includes:
- Tokenize protein sequences, add positional information, and contextualize residue embeddings with Transformer self-attention. section 12.3.3
- Let every sequence position attend globally, allowing the model to learn relationships between distant residues involved in folding or bonding. section 12.3
- Evaluate masked-token accuracy or pseudo-perplexity on held-out sequences while checking for near-duplicate leakage and training-set bias. section 12.3.3
- Reuse learned embeddings for downstream tasks such as antibody classification, or score mutations without additional training through zero-shot inference. section 12.4
Masked-residue prediction
Given MKTAYIA[MASK]QRQISF, the model predicts K at the masked position. A trained model can also rank likely amino acids, such as K (0.721), R (0.178), and Q (0.064), for a masked residue. section 12.3.3
Listing 12.1 Protein Language Model using Transformer Encoder
class ProteinLanguageModel(nn.Module):
def __init__(
self, vocab_size,
padding_idx, embedding_dim,
num_layers, num_heads,
ffn_dim, dropout,
):
super().__init__()
self.embedding = nn.Embedding( #A
vocab_size, embedding_dim, padding_idx #A
) #A
self.pe = PositionalEncoding(embedding_dim) #B
self.encoder = Encoder( #C
num_layers=num_layers, #C
embedding_dim=embedding_dim, #C
num_heads=num_heads, #C
ffn_dim=ffn_dim, #C
dropout=dropout, #C
) #C
self.lm_head = nn.Linear( #D
embedding_dim, vocab_size #D
) #D
def forward(self, batch):
input_ids = batch["input_ids"]
attention_mask = batch["attention_mask"]
x = self.embedding(input_ids) #E
x = self.pe(x) #F
x = self.encoder(x, attention_mask) #G
logits = self.lm_head(x) #H
return logitsInspect a compact PyTorch-style implementation containing the embedding layer, positional encoding, Transformer encoder, and language-model output head. section 12.3.3
