A validation set is held-out data used to measure model performance without updating model weights. It lets training track metrics such as accuracy and F1 separately from training results, while keeping the evaluation data outside parameter learning. section 2.4.1 section 2.4.4
Use validation data to:
- Compare training and held-out performance after each epoch, using separate result names and metric histories. section 2.4.1
- Evaluate a candidate architecture or hyperparameter setting, such as layer count, neurons per layer, or learning rate. section 5.4.2
- Protect the final evaluation from repeated reuse by creating temporary training/validation splits from the original training data. section 5.4.2
- Evaluate in model evaluation mode with torch.no_grad(), and choose a validation batch size for speed because it does not affect results. section 2.4.4 section 2.5
Tracking validation accuracy in PyTorch
Create a separate DataLoader for held-out examples and pass it to the training function as test_loader. After each training epoch, switch the model to evaluation mode, run the loader under torch.no_grad(), and append its accuracy and F1 values alongside the training metrics. Plot train Acc against validation Acc to inspect how performance changes over time. section 2.4.4
Figure 5.17 The four steps of defining an objective function that Optuna can use to optimize our neural network training. The two most important steps shown as code are getting the hyperparameters and computing the result!
The objective-function workflow shows where validation splitting and validation-score computation fit into Optuna hyperparameter optimization. section 5.4.2
