liveBook · Manning

Pseudo-perplexity

As explained in Build AI Drug Discovery Pipelines

Pseudo-perplexity is an evaluation metric for masked language models that aggregates the model's conditional uncertainty over tokens masked in a sequence.

Pseudo-perplexity (PPPL) summarizes how plausible a complete protein sequence is under a masked language model. With ESM-2, a lower PPPL means the sequence is more typical of the model’s training distribution. section 12.3.5

To score a protein sequence:

  • Mask one amino acid position and ask the model for the probability of the actual amino acid at that position. section 12.3.5
  • Repeat for every position, average the negative log-probabilities, and exponentiate the result to obtain PPPL. section 12.3.5
PPPL    = exp(-1N    × Σ log P(actual_aa_i    | context))

The equation combines the conditional probabilities for all positions into one sequence score. section 12.3.5

Combining antibody mutations

Suppose three antibody framework mutations have individually favorable log-likelihood ratios of +0.8, +1.2, and +0.6. Score the sequence containing all three and compare its PPPL with that of the wild-type sequence. A higher PPPL for the variant may signal unfavorable interactions among the mutations, despite their individual scores. section 12.3.5

PPPL measures statistical plausibility, not a guarantee of stability, folding, expression, binding, or biological function. Use it to prioritize combinations for testing, rather than as proof that a design will work. section 12.3.5 section 12.4

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains sequential masking, probability aggregation, interpretation, and limitations without providing a PPPL implementation.
Examples
Compares a three-mutation antibody variant with its wild-type sequence after scoring the mutations individually.
What you can do after
Evaluate/understand sequence plausibility when prioritizing protein variants for testing.
How it connects
Connects masked language modeling with log-likelihood ratios, mutation effect prediction, and epistasis.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.