liveBook · Manning

Validation

As explained in Grokking Machine Learning, Second Edition

Validation is the process of evaluating candidate models or hyperparameters on held-out data to guide model selection before final testing.

Validation evaluates candidate models on held-out data before final testing. It helps choose a model that learns useful patterns rather than memorizing its training examples. section 4.5 section 4.0

Use separate portions of the dataset for separate decisions:

  • Train every candidate model with the training set. section 4.5
  • Compare candidates or hyperparameter choices with validation performance, such as selecting among polynomial models or regularization values. section 4.11 section 4.5
  • Use the testing set only at the end to estimate how well the chosen model performs on unseen data. section 4.5

Choosing a polynomial model

Suppose degree-1, degree-2, and degree-10 polynomial regressions are trained on the same training data. Validation error can identify degree 2 as the best choice, while the untouched testing set provides the final check. Common splits include 60-20-20 or 80-10-10 for training, validation, and testing. section 4.5

Error = Regression error + λ Regularization term

The equation shows how validation can select the regularization parameter λ by comparing models that balance regression error and simplicity. section 4.11

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Defines a three-way holdout workflow with example splits, but does not extend beyond this validation-set procedure.
Examples
Uses polynomial degrees, regularization values, validation error, and 60-20-20 or 80-10-10 dataset splits.
What you can do after
Apply validation to select a model or hyperparameter before using the testing set once.
How it connects
Connects model selection with underfitting, overfitting, testing, polynomial regression, and regularization.

Also explained in

Compare how 44 books explain validation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.