Cross-validation

Q: How deeply does Delivering Machine Learning Projects: From design to deployment cover cross-validation?

Defines rotating train/test partitions, score aggregation, leave-one-out evaluation, and computational limits without developing algorithmic internals.

As explained in Delivering Machine Learning Projects: From design to deployment

Cross-validation is a resampling technique that repeatedly trains and evaluates a model on different partitions of available data.

Cross-validation estimates model performance by repeatedly training on most data partitions and testing on the remaining partition. Aggregating the results produces a cross-validated score while keeping each test partition outside that training run, reducing information leakage and unconscious optimization. section 8.2.1

Use a cross-validation ticket or evaluation checklist to:

Divide the dataset into disjoint partitions, such as ten folds. section 8.2.1
Train a separate model for each fold, holding that fold out for testing and using the others for training. section 8.2.1
Aggregate the fold-level results and compare relevant measures rather than relying on accuracy alone. section 8.2.1 section 8.6
Record the computational trade-off: cross-validation suits simple models and small or sparse datasets but becomes expensive for complex models and large datasets. section 8.2.1

Table 8.1 Generating a cross-validation set (view table figure)

	P1	P2	P3	P4	P5	P6	P7	P8
Set 1	Test	Train	Train	Train	Train	Train	Train	Train
Set 2	Train	Test	Train	Train	Train	Train	Train	Train
Set 3	Train	Train	Test	Train	Train	Train	Train	Train
Set 4	Train	Train	Train	Test	Train	Train	Train	Train
Set 5	Train	Train	Train	Train	Test	Train	Train	Train
Set 6	Train	Train	Train	Train	Train	Test	Train	Train
Set 7	Train	Train	Train	Train	Train	Train	Test	Train
Set 8	Train	Train	Train	Train	Train	Train	Train	Test

Use the fold matrix to verify that each partition serves as the test set once while the remaining partitions serve as training data. section 8.2.1

How Delivering Machine Learning Projects: From design to deployment covers it

Cover of Delivering Machine Learning Projects: From design to deployment

liveBook manning.com

Read Delivering Machine Learning Projects: From design to deployment on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Ensemble Methods for Machine LearningK-fold cross-validation partitions data into k folds and repeatedly trains and evaluates a model using one fold for validation and the others for training.
liveBook manning.com

Causal Inference for Data ScienceCross-validation is a model-evaluation procedure that repeatedly partitions data into training and validation portions to estimate predictive performance.
liveBook manning.com

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AICross-validation is a model-evaluation technique that repeatedly partitions data into training and validation subsets.
liveBook manning.com

Effective Conversational AIK-fold cross-validation is a model-evaluation technique that partitions a dataset into k folds and repeatedly evaluates a model with each fold held out in turn.
liveBook manning.com

Deep Learning with R, Third EditionK-fold cross-validation is an evaluation procedure that partitions data into K subsets and repeatedly trains on K−1 subsets while evaluating on the remaining subset.
liveBook manning.com

Deep Learning with Python, Third EditionA validation procedure that partitions data into K folds, trains on K−1 folds, and evaluates on the remaining fold in rotation.
liveBook manning.com

Grokking StatisticsCross-validation repeatedly partitions data into complementary training and evaluation folds to assess model performance.
liveBook manning.com

Causal AICross-validation is a model evaluation technique that assesses predictive performance on data withheld from training.
liveBook manning.com

Building LLM Applications with DSPyCross-validation is an evaluation technique that repeatedly partitions data into training and validation subsets to produce a more reliable estimate.
liveBook manning.com

Grokking Machine LearningA resampling technique that partitions a dataset into k portions and repeatedly trains on k−1 portions while validating on the remaining portion, then averages the validation scores.
liveBook manning.com

Financial AI in PracticeCross-validation is a model evaluation technique that partitions data into subsets and repeatedly trains and evaluates a model on different subsets.

Compare how 12 books explain cross-validation →

Related concepts

covariateuses · A covariate is an observed or supplied variable included in a statistical, causal, or predictive model to describe, explain, predict, or adjust for variation in an outcome.
grid searchused by · Grid search is a search method that evaluates configurations formed from specified candidate values, typically every combination in their Cartesian product.
training datasetuses · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.

This explainer was generated by Manning's knowledge graph from the text of Delivering Machine Learning Projects: From design to deployment. The section links point to the parts of the book it draws on.