liveBook · Manning

Validation set

As explained in Timeless Algorithms, The Foundational Ideas

Held-out data used to assess model performance and guide model selection during development.

A model can perform well on its training observations yet fail to generalize. A validation set—observations excluded from parameter fitting—helps compare candidate models and guide development decisions. A separate test set is reserved for final evaluation after those decisions are made. section 11.6.1 section 11.6.2

The three data roles differ by the decisions they support:

  • Training data estimate model parameters, such as weights and biases. section 11.6.2
  • Validation data guide architecture, hyperparameter, and early-stopping decisions. section 11.6.1 section 11.6.2
  • Test data provide a final evaluation after selection; validation success does not prove that the best possible model was found. section 11.6.1

Selecting an SVM penalty

In k-fold cross-validation, each fold takes a turn as validation data while the model is fitted on the other folds. Repeat this for each candidate value of C, average the validation scores, and generally select the value with the strongest average performance. The choice of score—and any preference among similarly performing values—depends on the application. section 8.3.5

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Distinguishes training, validation, and test roles, then details fold-based SVM evaluation without guaranteeing an optimal model.
Examples
Compares neural-network configurations and candidate SVM penalty values using held-out observations or folds.
What you can do after
Apply it to compare candidate settings and reserve an independent test set for final evaluation.
How it connects
Links model selection and early stopping to overfitting, regularization, and cross-validation.

Also explained in

Compare how 6 books explain validation set →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.