liveBook · Manning

Evaluation dataset

As explained in Timeless Algorithms, The Foundational Ideas

A test set is data kept separate from model fitting and selection for estimating the performance of a completed model.

A test set is reserved to estimate how a completed model performs beyond the data used to fit it. Training data estimate parameters; validation data guide choices such as architecture, hyperparameters, and early stopping; test data are held back until those choices are finished. section 11.6.2 section 11.6.1

Keeping test observations out of model selection matters: using them to choose settings can make the resulting performance estimate overly optimistic. Strong test performance supports conclusions only for conditions represented in the test data; different populations, environments, or measurement processes may limit those conclusions. section 8.3.5 section 11.6.2

Selecting an SVM penalty

Cross-validation selects the penalty parameter C using training data. The SVM is then refitted with that value, and the untouched test set is used once to estimate generalization. section 8.3.5

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Distinguishes fitting, validation-based selection, and final testing, but does not establish performance under unrepresented conditions.
Examples
Uses an SVM after penalty selection and a neural network after architecture selection as test-set examples.
What it assumes
Assumes training and validation observations have supported fitting and model selection before final testing.
What you can do after
Evaluate a completed model once on untouched observations without using them to choose its settings.
How it connects
Connects training data, validation data, cross-validation, model selection, and generalization.

Also explained in

Compare how 41 books explain evaluation dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.