liveBook · Manning

Validation

As explained in Grokking Machine Learning

A model-development procedure that evaluates candidate models or settings on held-out data without using the final test data for those decisions.

How can you choose a model without letting your final test data influence that choice? Set aside validation data for comparing candidate models or settings. Keep the test set untouched until the end, so it can give a final check of model performance. section 4.3

Give each split a distinct job:

  • Train candidate models on the training set. section 4.3
  • Use the validation set to choose which model or settings to use. section 4.3
  • Use the test set only at the end to check how well the selected model did. section 4.3

Choosing a polynomial model

Train polynomial regression models of degrees 1, 2, and 10. Compare their validation errors to choose a model; reserve the test set for the final check. Common splits include 60% training, 20% validation, 20% testing, or 80%, 10%, and 10%. section 4.3

How Grokking Machine Learning covers it

How deep it goes
Separates data into training, validation, and testing roles, gives common split ratios, and defers further procedural detail.
Examples
Compares polynomial regression models of degrees 1, 2, and 10, and gives 60-20-20 and 80-10-10 dataset splits.
What you can do after
Apply the three-way split to choose a model with validation data and reserve the test data for final evaluation.
How it connects
Connects validation with model selection, hyperparameters, overfitting, grid search, and k-fold cross-validation.
Cover of Grokking Machine Learning
liveBookmanning.com
Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

Compare how 30 books explain validation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.