liveBook · Manning

Evaluation dataset

As explained in Grokking Machine Learning, Second Edition

A held-out subset of data used to evaluate models or choose training settings without fitting those settings to the evaluation examples.

A validation set is a held-out portion of data used to make choices about a model without using the final testing data. It helps choose among models or hyperparameters while keeping the testing set untouched until the end. section 4.5 section 4.15

Keep the three dataset roles separate:

  • Training set: fit the models. section 4.5
  • Validation set: choose which model or settings to use. section 4.5
  • Testing set: check the final model’s performance once decisions are finished. section 4.5

Choosing polynomial complexity

Suppose three polynomial regression models have degrees 1, 2, and 10. Train all three on the training set, compare their validation errors, and choose the model with the best validation performance. Model 2 is selected in the example; the testing set is then reserved for the final check. Common splits include 60-20-20 or 80-10-10 for training, validation, and testing. section 4.5

Selecting regularization

Try candidate values such as 10, 1, 0.1, and 0.01 for λ, then select the value producing the best validation performance. The validation set helps control model complexity without spending the testing set on repeated decisions. section 4.11

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Explains three-way data splitting, model selection, and typical proportions, but does not detail implementation procedures.
Examples
Uses polynomial models of degrees 1, 2, and 10, plus candidate regularization values and 60-20-20 or 80-10-10 splits.
What it assumes
Builds on training and testing sets while defining the additional validation role and its separation from final evaluation.
What you can do after
Evaluate models and choose complexity or regularization settings without consuming the testing set.
How it connects
Connects training sets, testing sets, hyperparameters, overfitting, underfitting, model complexity, and regularization.

Also explained in

Compare how 41 books explain evaluation dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.