liveBook · Manning

Evaluation dataset

As explained in Build AI Drug Discovery Pipelines

A validation set is a held-out collection of examples used during development to assess model performance and guide configuration choices.

An evaluation dataset contains examples held out from fitting a model so its predictions can be assessed on data it has not trained on. A validation set is used during development to compare models or guide training choices; a separate test set is saved for evaluating the final model. section 3.3.3

This separation helps assess generalization: performance on new examples rather than memorized training examples. In drug discovery, generalization is especially important because labeled compounds are expensive and new molecules may differ structurally from those used for training. section 1.2.2 section 3.3.3

A practical validation workflow:

  • Keep a test set aside, then divide the remaining training data into smaller training and validation sets. Fit candidate models on the smaller training set and compare their validation performance. section 3.3.3
  • Consider k-fold cross-validation when a single split leaves too little data for training or validation. Divide the training set into k folds, train k models on k−1 folds each, and validate each on the remaining fold. Average the validation scores. section 3.3.3
  • After selecting a model, retrain it on all training data and evaluate it on the held-out test set. Repeatedly consulting the test set during development can contaminate the assessment. section 3.3.3

The split itself matters. A random split can put nearly identical molecular analogs in both training and validation folds, making scores look better than real-world performance. A small validation set can also give an unreliable approximation of generalization. section 3.3.3

Listing 3.9 5-fold cross validation
>>> from sklearn.model_selection import cross_validate
>>> scoring = {'acc': 'accuracy'}
>>> lin_cls_scores = cross_validate(
        lin_cls, train_fingerprints, train_labels, scoring=scoring, cv=5
    )
>>>lin_cls_scores["test_acc"].mean()
0.7933463161311263

This scikit-learn example evaluates a classifier with five-fold cross-validation and averages its validation accuracy, yielding approximately 0.793. section 3.3.3

Monitoring neural-network training

A validation pass measures loss on held-out data after training batches. In one run, training loss falls from 3.7013 to 0.2256 over 50 epochs, while validation loss reaches 0.7483 by epoch 5 and is 0.6720 at epoch 50. A validation-loss plateau can motivate fewer epochs, early stopping, or learning-rate adjustments. section 8.2.3

Listing 8.7 Train the Model
def train_model(
    model, train_loader, val_loader, criterion, optimizer, 
    scheduler, device, num_epochs=10
):
    model.train()    #A
    train_losses = []    #B
    val_losses = []
    
    for epoch in range(num_epochs):    #C
        running_loss = 0.0    #D
        for inputs, targets in train_loader:
            inputs, targets = inputs.to(device), targets.to(device)
            
            optimizer.zero_grad()    #E
            outputs = model(inputs)    #F
            loss = criterion(outputs, targets)    #G
            loss.backward()    #H
            optimizer.step()    #I
            
            running_loss += loss.item()
        
        train_loss = running_loss / len(train_loader)
        train_losses.append(train_loss)
        
        model.eval()    #J
        val_loss = 0.0
        with torch.no_grad():    #K
            for inputs, targets in val_loader:
                inputs, targets = inputs.to(device), targets.to(device)
                outputs = model(inputs)
                loss = criterion(outputs, targets)
                val_loss += loss.item()
        
        val_loss /= len(val_loader)
        val_losses.append(val_loss)
        
        print(f'Epoch {epoch+1}/{num_epochs}', 
              f'Train Loss: {train_loss:.4f}',
              f'Val Loss: {val_loss:.4f}')    #L
        
        scheduler.step(val_loss)    #M
        model.train()    #N

The training loop uses separate data loaders and evaluates validation batches in evaluation mode without gradient calculation. section 8.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains validation splits, k-fold cross-validation, and epoch-level validation, while deferring detailed splitting-strategy selection.
Examples
Illustrates five-fold classifier accuracy and neural-network training and validation losses across 50 epochs.
What you can do after
Applies validation with scikit-learn cross-validation and a PyTorch neural-network training loop.
How it connects
Connects validation sets to generalization, overfitting, cross-validation, and test-set contamination.

Also explained in

Compare how 41 books explain evaluation dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.