A test set is reserved to estimate how a completed model performs beyond the data used to fit it. Training data estimate parameters; validation data guide choices such as architecture, hyperparameters, and early stopping; test data are held back until those choices are finished. section 11.6.2 section 11.6.1
Keeping test observations out of model selection matters: using them to choose settings can make the resulting performance estimate overly optimistic. Strong test performance supports conclusions only for conditions represented in the test data; different populations, environments, or measurement processes may limit those conclusions. section 8.3.5 section 11.6.2
Selecting an SVM penalty
Cross-validation selects the penalty parameter C using training data. The SVM is then refitted with that value, and the untouched test set is used once to estimate generalization. section 8.3.5








































