An evaluation dataset, called a validation dataset during model selection, lets you compare candidate configurations by their performance on data beyond the training examples. The aim is to choose a model that explains more than its training data and generalizes to unseen examples. Keep its selection role distinct from the test data used to assess performance on new, unseen examples. section 7.4 section 1.1
Use validation performance to search a hyperparameter space:
- Grid search exhaustively tries specified combinations. Values such as 0.1, 1, 10, and 100 can probe different orders of magnitude; this approach suits smaller search spaces. section 7.4
- Random search samples hyperparameters from chosen distributions. Its iteration count need not grow with the number of parameters. section 7.4
Choosing an SVM configuration
Try candidate kernels, kernel parameters, and regularization constants, then favor the configuration with the strongest validation performance. For a continuous response, one possible loss is mean squared error: MSE = (1/n) Σᵢ(yᵢ − ŷᵢ)². It averages squared prediction errors, giving larger deviations a larger penalty. section 7.4 section 1.1
