liveBook · Manning

Protein language model

As explained in Build AI Drug Discovery Pipelines

A protein language model is a model trained on protein sequences to learn sequence representations or predict biologically meaningful sequence patterns.

A protein language model (PLM) learns biological patterns from unlabeled amino-acid sequences. Masked language modeling hides residues and trains the model to predict them, producing contextual representations that capture sequence constraints, structural tendencies, and long-range dependencies. section 12.3.3 section 12.4

A practical PLM workflow includes:

  • Tokenize protein sequences, add positional information, and contextualize residue embeddings with Transformer self-attention. section 12.3.3
  • Let every sequence position attend globally, allowing the model to learn relationships between distant residues involved in folding or bonding. section 12.3
  • Evaluate masked-token accuracy or pseudo-perplexity on held-out sequences while checking for near-duplicate leakage and training-set bias. section 12.3.3
  • Reuse learned embeddings for downstream tasks such as antibody classification, or score mutations without additional training through zero-shot inference. section 12.4

Masked-residue prediction

Given MKTAYIA[MASK]QRQISF, the model predicts K at the masked position. A trained model can also rank likely amino acids, such as K (0.721), R (0.178), and Q (0.064), for a masked residue. section 12.3.3

Listing 12.1 Protein Language Model using Transformer Encoder
class ProteinLanguageModel(nn.Module):
    def __init__(
        self, vocab_size,
        padding_idx, embedding_dim,
        num_layers, num_heads,
        ffn_dim, dropout,
    ):
        super().__init__()
 
        self.embedding = nn.Embedding(    #A
            vocab_size, embedding_dim, padding_idx    #A
        )    #A
 
        self.pe = PositionalEncoding(embedding_dim)    #B
 
        self.encoder = Encoder(    #C
            num_layers=num_layers,    #C
            embedding_dim=embedding_dim,    #C
            num_heads=num_heads,    #C
            ffn_dim=ffn_dim,    #C
            dropout=dropout,    #C
        )    #C
 
        self.lm_head = nn.Linear(    #D
            embedding_dim, vocab_size    #D
        )    #D
 
    def forward(self, batch):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        x = self.embedding(input_ids)    #E
        x = self.pe(x)    #F
        x = self.encoder(x, attention_mask)    #G
        logits = self.lm_head(x)    #H
        return logits

Inspect a compact PyTorch-style implementation containing the embedding layer, positional encoding, Transformer encoder, and language-model output head. section 12.3.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Builds a small masked-language model and explains global attention, evaluation, leakage, and transfer, without detailing full-scale training systems.
Examples
Demonstrates masked-residue prediction, attention heatmaps, antibody classification, and ESM-2 embeddings supplied to a structure-generation pipeline.
What you can do after
Builds a small PLM, evaluates masked predictions, transfers embeddings to classification, and screens mutations through zero-shot sequence scoring.
How it connects
Connects masked language modeling with Transformers, self-attention, protein structure prediction, antibody classification, and zero-shot mutation effects.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.