liveBook · Manning

Masked language modeling

As explained in Build AI Drug Discovery Pipelines

Masked language modeling is a self-supervised technique that trains a model to predict tokens hidden or replaced within an input sequence.

Masked language modeling (MLM) is a self-supervised technique: a model predicts tokens hidden in an input sequence, using the original sequence as its training signal. For proteins, the tokens represent amino acids, and no external labels are needed. section 12.3.3

Predicting a missing amino acid

Given the protein sequence MKTAYIAKQRQISF, hide the K to produce MKTAYIA[MASK]QRQISF. The model uses the surrounding sequence to predict the missing amino acid. In one prediction, it assigned K a probability of 0.721, followed by R at 0.178 and Q at 0.064. section 12.3.3

Protein sequences are not random strings: evolution favors patterns compatible with stable, functional structures. Predicting missing residues encourages a model to learn recurring patterns, residues that appear together, and relationships between distant positions. The resulting representations can support downstream tasks, such as classifying antibodies; MLM alone does not make that classification decision. section 12.3.3 section 12.4

A small protein MLM follows this workflow:

  • Start with unlabeled protein sequences and hide selected amino acids. Their original identities provide the targets for prediction. section 12.3.3
  • Convert tokens into embeddings, add positional information, and use a transformer encoder's self-attention to place each residue in context. A prediction head produces a distribution over amino acids at each position. section 12.3.3
  • Evaluate masked-token accuracy or pseudo-perplexity on held-out sequences. Check for near-duplicate sequences across the split, which can leak information into evaluation. section 12.3.3
Listing 12.1 Protein Language Model using Transformer Encoder
class ProteinLanguageModel(nn.Module):
    def __init__(
        self, vocab_size,
        padding_idx, embedding_dim,
        num_layers, num_heads,
        ffn_dim, dropout,
    ):
        super().__init__()
 
        self.embedding = nn.Embedding(    #A
            vocab_size, embedding_dim, padding_idx    #A
        )    #A
 
        self.pe = PositionalEncoding(embedding_dim)    #B
 
        self.encoder = Encoder(    #C
            num_layers=num_layers,    #C
            embedding_dim=embedding_dim,    #C
            num_heads=num_heads,    #C
            ffn_dim=ffn_dim,    #C
            dropout=dropout,    #C
        )    #C
 
        self.lm_head = nn.Linear(    #D
            embedding_dim, vocab_size    #D
        )    #D
 
    def forward(self, batch):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        x = self.embedding(input_ids)    #E
        x = self.pe(x)    #F
        x = self.encoder(x, attention_mask)    #G
        logits = self.lm_head(x)    #H
        return logits

Inspect the PyTorch model components: token embeddings, positional encoding, a transformer encoder, and a prediction head. section 12.3.3

The same idea scales to ESM-2, an encoder-only transformer family trained on roughly 65 million clustered protein sequences. Its learned representations can support applications such as variant-effect prediction and secondary-structure prediction. Training data biases remain a limitation, so useful predictions still require task-appropriate evaluation. section 12.3.5 section 13.2.1 section 12.3.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains masked-residue prediction and implements a small transformer model, then relates the approach to scaled ESM-2 models.
Examples
Uses a masked lysine prediction and a small protein language model, alongside ESM-2 and antibody classification.
What you can do after
Builds a small protein language model and evaluates masked-token predictions on held-out sequences.
How it connects
Connects self-supervised learning, transformer self-attention, protein embeddings, transfer learning, and antibody classification.

Also explained in

Compare how 6 books explain masked language modeling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.