A model can perform well on its training observations yet fail to generalize. A validation set—observations excluded from parameter fitting—helps compare candidate models and guide development decisions. A separate test set is reserved for final evaluation after those decisions are made. section 11.6.1 section 11.6.2
The three data roles differ by the decisions they support:
- Training data estimate model parameters, such as weights and biases. section 11.6.2
- Validation data guide architecture, hyperparameter, and early-stopping decisions. section 11.6.1 section 11.6.2
- Test data provide a final evaluation after selection; validation success does not prove that the best possible model was found. section 11.6.1
Selecting an SVM penalty
In k-fold cross-validation, each fold takes a turn as validation data while the model is fitted on the other folds. Repeat this for each candidate value of C, average the validation scores, and generally select the value with the strongest average performance. The choice of score—and any preference among similarly performing values—depends on the application. section 8.3.5





