liveBook · Manning

Cold-start evaluation

As explained in Build AI Drug Discovery Pipelines

Cold-start evaluation tests a model on entities or entity types that are absent from its training data.

Cold-start evaluation tests whether a drug–target affinity model generalizes to entities absent from training data. Unlike warm-start evaluation, it prevents the model from seeing the same compound or protein in training and testing, making genuinely new chemistry or targets harder to predict. section 11.1.1 section 11.5.5

Use distinct holdout designs to control which entities remain unseen:

  • Cold-drug splits hold out entire drugs, so every test drug is absent from training. section 11.5.5
  • Cold-target splits hold out entire proteins, so every test protein is absent from training. section 11.5.5
  • Cold-drug/cold-target splits hold out both partners in every test pair. section 11.5.5
  • Compare these results with warm-start scores cautiously, because repeated exposure to compounds and proteins can support memorization and overstate performance. section 11.1.1 section 11.5.5

KIBA provides a realistic motivation: its sparse interaction matrix resembles incomplete experimental data, while its diverse compounds make prediction for genuinely new chemical space difficult. Cold-start evaluation therefore tests a stricter form of generalization than predicting unseen combinations of familiar drugs and proteins. section 11.1.1 section 11.5.5

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines three holdout variants and contrasts them with warm-start evaluation, but does not specify metrics or implementation code.
Examples
Uses KIBA and DAVIS drug–target datasets, plus cold-drug, cold-target, and combined holdout designs.
What it assumes
Assumes familiarity with drug–target pairs, training and test sets, and model generalization.
What you can do after
Evaluate affinity-model generalization by selecting holdouts that exclude drugs, proteins, or both from training.
How it connects
Connects data splitting, validation strategy, warm-start evaluation, memorization, and generalization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.