liveBook · Manning

Cross-validation

As explained in Timeless Algorithms, The Foundational Ideas

Cross-validation is a model evaluation technique that repeatedly withholds portions of data from fitting and assesses performance on those held-out observations.

Training fit alone cannot show whether a model will generalize. Cross-validation estimates performance on observations withheld from fitting, allowing candidate penalty values to be compared without evaluating them on their training data. section 8.3.4 section 8.3.5

In k-fold cross-validation, divide the data into k approximately equal folds and repeat these steps for each candidate value of C:

  • Hold out one fold for validation and fit the model on the other k − 1 folds. section 8.3.5
  • Evaluate the fitted model on the held-out fold using a suitable classification measure. section 8.3.5
  • Rotate the held-out fold until every fold has served as validation data, then average the results. section 8.3.5
  • Compare candidate values of C by their average validation performance. section 8.3.5

Five folds may suit larger data sets, while ten may be preferable with less data because each fit uses 90 rather than 80 percent of observations. Choose k with class balance, estimate variability, and computational cost in mind; each fold needs enough observations from both classes. Accuracy is one option, but balanced accuracy, precision, recall, or F1 may better suit imbalanced classes or unequal error consequences. section 8.3.5

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines k-fold evaluation through model fitting, held-out scoring, fold rotation, and averaging, while leaving k to practical judgment.
Examples
Compares five-fold and ten-fold choices and names accuracy, balanced accuracy, precision, recall, and F1 as evaluation measures.
What it assumes
Assumes candidate penalty values and class-labeled observations, while defining folds and the evaluation procedure.
What you can do after
Apply fold-based held-out evaluation to compare candidate SVM penalty values by average validation performance.
How it connects
Connects soft-margin support vector machines, classification measures, and model generalization.

Also explained in

Compare how 15 books explain cross-validation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.