liveBook · Manning

Cross-validation

As explained in Build AI Drug Discovery Pipelines

Cross-validation estimates model performance or selects configurations by repeatedly training and evaluating on different partitions of the training data.

Cross-validation estimates how well a model generalizes without repeatedly reusing the held-out test set. In k-fold cross-validation, split the training data into k folds, train k models on k−1 folds, and validate each model on the remaining fold. Average the validation scores, then retrain the selected model on all training data before final test evaluation. section 3.3.3

Use cross-validation to compare models and tune configurations while preserving a clean final evaluation:

  • Keep the test set sealed until the final model is chosen; repeated test-set use contaminates its estimate of unseen-data performance. section 3.3.3 section 3.4.2
  • Choose folds carefully: random folds can place near-identical molecular analogs in both training and validation data, inflating apparent performance. section 3.3.3
  • Report metrics suited to the task, such as accuracy, macro-precision, macro-recall, F1, or area under the precision–recall curve. section 3.3.3 section 6.4.1
  • Use stratified folds when class representation matters, and average fold scores rather than trusting one partition. section 6.4.1 section 3.3.3

Five-fold molecular classification

With Scikit-Learn, call cross_validate(lin_cls, train_fingerprints, train_labels, cv=5) and average the returned validation accuracies. A five-fold run trains five models, each using four-fifths of the training fingerprints and validating on the remaining fifth; the reported mean accuracy is 0.7933463161311263. A dummy most-frequent classifier provides a baseline accuracy of 0.6938775510204082. section 3.3.3

Figure 3.13 k-fold cross validation to evaluate model candidates on training data split into train and validation folds. The best model is retrained on all training data and evaluated on the held-out test data. Not shown, but before deploying the best model to a production environment, we would also retrain it on all available data, including the test set.

The fold diagram shows model candidates evaluated on training folds, the selected model retrained on all training data, and the untouched test evaluation. section 3.3.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains fold construction, averaging, metrics, leakage, and retraining, but defers production deployment and maintenance.
Examples
Demonstrates Scikit-Learn fingerprints, five-fold accuracy, a dummy-classifier baseline, stratified molecular classification, and graph-network validation.
What it assumes
Assumes training and test sets, supervised models, and evaluation metrics, while defining folds and validation roles explicitly.
What you can do after
Apply it via Scikit-Learn to compare models, tune configurations, calculate metrics, and select a candidate without opening the test set.
How it connects
Connects model selection with training-validation splits, hyperparameter tuning, overfitting, generalization error, and test-set contamination.

Also explained in

Compare how 15 books explain cross-validation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.