A validation set is held-out data used to compare models and make choices such as which model or hyperparameters to use. Keep it separate from training data, and reserve the test set for a final check after those choices are made. section 4.3 section 4.7
A three-way split gives each portion a distinct job:
- Training set: fit the models. section 4.3
- Validation set: compare models and guide choices. section 4.3
- Test set: check the selected model at the end; do not use it to make model decisions. section 4.3
Common splits include 60% training, 20% validation, and 20% testing, or 80%, 10%, and 10%. These proportions are examples, not fixed rules. section 4.3
Choosing a model
For polynomial regression, compare degree-1, degree-2, and degree-10 models using validation error; use the test set only after choosing a model. In a Titanic example, candidate classifiers are evaluated on validation data before a selected model is checked on test data. section 4.3 section 13.4
