Generalization is a model’s ability to make useful predictions on previously unseen data from the relevant distribution. A model that memorizes training examples rather than learning broader patterns may perform poorly on new inputs; this is overfitting. section 1.2.2
In drug discovery, generalization matters because a model may encounter compounds unlike those used for training. Labeled compounds are expensive to obtain, and chemical space is vast, making a large, diverse training set difficult to assemble. section 1.2.2
A practical evaluation workflow separates model selection from the final test:
- Hold out a test set to estimate performance on unseen data. Avoid repeatedly using it to make modeling decisions, because reuse contaminates the test. section 3.3.3
- Use k-fold cross-validation on the training data to compare candidates. Train each candidate on k−1 folds and validate it on the remaining fold, rotating the held-out fold and averaging the scores. section 3.3.3
- Choose the split with the intended application in mind. Random folds can put near-identical molecular analogs in both training and validation, making scores look better than real-world performance. section 3.3.3
Listing 3.9 5-fold cross validation
>>> from sklearn.model_selection import cross_validate
>>> scoring = {'acc': 'accuracy'}
>>> lin_cls_scores = cross_validate(
lin_cls, train_fingerprints, train_labels, scoring=scoring, cv=5
)
>>>lin_cls_scores["test_acc"].mean()
0.7933463161311263This Scikit-Learn example evaluates a linear classifier with five-fold cross-validation and reports a mean accuracy of about 0.793. section 3.3.3
Compare training and validation performance when diagnosing overfitting. A large gap can indicate that the model fits training data more closely than it predicts held-out data. Conversely, high training error can indicate underfitting: the model is too simple to learn the available patterns. section 4.4.3 section 8.2.3
Validation curves plot performance against a hyperparameter, helping identify a useful balance between underfitting and overfitting. For example, regularization restricts model weights and can reduce overfitting; polynomial feature degree controls the complexity of a nonlinear transformation. section 3.5 section 4.4.3
Monitoring neural-network training
A neural network’s validation loss can level off even while training loss continues to fall. In one run, training loss fell to 0.2256 by epoch 50, while validation loss was 0.6720. That pattern suggests trying fewer epochs or early stopping and adjusting the learning rate or scheduler. section 8.2.3


















