liveBook · Manning

Cross-validation

Cross-validation is a model-evaluation technique that repeatedly partitions available data into training and held-out subsets to estimate performance on unseen data.

How 15 Manning books explain it

Ensemble Methods for Machine Learning

K-fold cross-validation partitions data into k folds and repeatedly trains and evaluates a model using one fold for validation and the others for training.

How it is used
Evaluates models, simulates out-of-sample testing, and selects hyperparameters such as learning rate.
How deep it goes
Explains fold construction, repeated training and validation, stratification, averaging, refitting, and out-of-fold stacking features.
Examples
Uses five-fold stacking, breast-cancer learning-rate searches, digit-classification grid search, and CatBoost credit-approval tuning.
What you can do after
Apply it via scikit-learn, LightGBM, XGBoost, or CatBoost to compare settings and refit selected models.

Read how Ensemble Methods for Machine Learning explains cross-validation →

Causal Inference for Data Science

Cross-validation is a model-evaluation procedure that repeatedly partitions data into training and validation portions to estimate predictive performance.

How it is used
Evaluates and selects supervised predictive models before they are used on future, historically similar data.
How deep it goes
Explains predictive-model selection and its causal limitations, but does not detail fold construction or scoring procedures.
Examples
Uses house inspection versus earthquake resistance and Pac-Man level regressions with bonuses, ghosts, and completion time.
What you can do after
Evaluate predictive models by held-out capability while avoiding cross-validation alone as a basis for causal-model choice.

Read how Causal Inference for Data Science explains cross-validation →

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

Cross-validation is a model-evaluation technique that repeatedly partitions data into training and validation subsets.

How it is used
Estimates expected performance on future tabular data and supports model comparison before final fitting.
How deep it goes
Explains repeated holdouts, K-fold aggregation, stratification, prediction-based diagnostics, and tuning-oriented validation schemes.
Examples
Uses synthetic logistic classification, Airbnb and real-estate data, stratified neighborhood folds, and linear-regression prediction diagnostics.
What you can do after
Apply it via Scikit-learn to evaluate pipelines, compare models, inspect held-out predictions, and select validation-aware hyperparameters.

Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains cross-validation →

Build AI Drug Discovery Pipelines

Cross-validation estimates model performance or selects configurations by repeatedly training and evaluating on different partitions of the training data.

How it is used
Uses training data efficiently to compare models, tune configurations, and estimate generalization before final test evaluation.
How deep it goes
Explains fold construction, averaging, metrics, leakage, and retraining, but defers production deployment and maintenance.
Examples
Demonstrates Scikit-Learn fingerprints, five-fold accuracy, a dummy-classifier baseline, stratified molecular classification, and graph-network validation.
What you can do after
Apply it via Scikit-Learn to compare models, tune configurations, calculate metrics, and select a candidate without opening the test set.

Read how Build AI Drug Discovery Pipelines explains cross-validation →

Effective Conversational AI

K-fold cross-validation is a model-evaluation technique that partitions a dataset into k folds and repeatedly evaluates a model with each fold held out in turn.

How it is used
Measures a chatbot intent classifier's baseline performance when additional annotated test data is unavailable.
How deep it goes
Outlines repeated holdout, scoring, and reintegration, but leaves detailed testing instructions to further guidance.
Examples
Uses a 125-question chatbot baseline with 84% accuracy to show how sparse intents can produce zero recall.
What you can do after
Evaluate baseline classifier performance and identify weak intents while treating results cautiously.

Read how Effective Conversational AI explains cross-validation →

Deep Learning with R, Third Edition

K-fold cross-validation is an evaluation procedure that partitions data into K subsets and repeatedly trains on K−1 subsets while evaluating on the remaining subset.

How it is used
Guides model evaluation and training-duration choices when hold-out validation is too variable because little data is available.
How deep it goes
Explains partitioning, repeated training, score averaging, validation histories, plotting, and final-model training without formal derivation.
Examples
Demonstrates three- and four-fold splits, validation MAE histories, score plots, and a final regression model.
What you can do after
Apply it via an R training loop to estimate validation performance, compare epochs, and select a final training duration.

Read how Deep Learning with R, Third Edition explains cross-validation →

Timeless Algorithms, The Foundational Ideas

Cross-validation is a model evaluation technique that repeatedly withholds portions of data from fitting and assesses performance on those held-out observations.

How it is used
Selects a soft-margin SVM penalty value by estimating how candidate models perform on withheld observations.
How deep it goes
Defines k-fold evaluation through model fitting, held-out scoring, fold rotation, and averaging, while leaving k to practical judgment.
Examples
Compares five-fold and ten-fold choices and names accuracy, balanced accuracy, precision, recall, and F1 as evaluation measures.
What you can do after
Apply fold-based held-out evaluation to compare candidate SVM penalty values by average validation performance.

Read how Timeless Algorithms, The Foundational Ideas explains cross-validation →

Deep Learning with Python, Third Edition

A validation procedure that partitions data into K folds, trains on K−1 folds, and evaluates on the remaining fold in rotation.

How it is used
Stabilizes validation-based model comparison when limited data makes a single hold-out score unreliable.
How deep it goes
Implements fold construction, repeated training, score averaging, per-epoch histories, plotting, and practical epoch selection.
Examples
Uses four-fold scalar regression, validation MAE histories, epoch plots, and a final 130-epoch model.
What you can do after
Apply it via a model-training loop to obtain a more reliable validation estimate and guide training duration.

Read how Deep Learning with Python, Third Edition explains cross-validation →

Vibe Engineering

Cross-validation is a model-evaluation technique that repeatedly trains and evaluates a model on different partitions of a dataset.

How it is used
Checks whether unusually high model scores hold across data partitions when overfitting is a concern.
How deep it goes
Explains five-fold training and testing and reports fold accuracies, while linking to the implementation.
Examples
Uses a medicine-risk model with near-perfect precision–recall curves and reports five fold accuracies.
What you can do after
Evaluate and understand fold-to-fold accuracy variation as a check on model performance.

Read how Vibe Engineering explains cross-validation →

Grokking Statistics

Cross-validation repeatedly partitions data into complementary training and evaluation folds to assess model performance.

How it is used
Uses withheld data to check whether classification models generalize to unseen cases.
How deep it goes
Demonstrates three-fold iteration and prediction, but stops short of detailed statistical theory or advanced validation metrics.
Examples
Uses a toddler-meltdown classification dataset and a three-fold KFold implementation with scikit-learn.
What you can do after
Apply it via scikit-learn’s KFold to evaluate logistic-regression predictions on repeatedly held-out data.

Read how Grokking Statistics explains cross-validation →

Delivering Machine Learning Projects: From design to deployment

Cross-validation is a resampling technique that repeatedly trains and evaluates a model on different partitions of available data.

How it is used
Supports model testing and selection when data is scarce or a simple train/test split would waste available training data.
How deep it goes
Defines rotating train/test partitions, score aggregation, leave-one-out evaluation, and computational limits without developing algorithmic internals.
Examples
Uses ten-fold evaluation, an eight-partition P1–P8 matrix, and leave-one-out evaluation across 1,000 examples.
What you can do after
Evaluate model performance by rotating held-out partitions, aggregating scores, and documenting measurement trade-offs for selection.

Read how Delivering Machine Learning Projects: From design to deployment explains cross-validation →

Causal AI

Cross-validation is a model evaluation technique that assesses predictive performance on data withheld from training.

How it is used
Compares models by estimating predictive performance on data excluded from training.
How deep it goes
Defines held-out prediction and contrasts it with goodness-of-fit, but does not specify fold construction or computational procedure.
Examples
Uses a model with strong training-data fit but poor withheld-data prediction as the central comparison.
What you can do after
Evaluate models by distinguishing generalization to unseen data from fit to training data.

Read how Causal AI explains cross-validation →

Building LLM Applications with DSPy

Cross-validation is an evaluation technique that repeatedly partitions data into training and validation subsets to produce a more reliable estimate.

How it is used
Estimates model skill more reliably than a single train-validation-test split.
How deep it goes
Explains repeated training-validation splits, their reliability benefit, and their cost without detailing implementation.
What you can do after
Evaluate model skill using repeated splits and weigh greater reliability against added cost.

Read how Building LLM Applications with DSPy explains cross-validation →

Grokking Machine Learning

A resampling technique that partitions a dataset into k portions and repeatedly trains on k−1 portions while validating on the remaining portion, then averages the validation scores.

How it is used
Reuses data for training and validation when evaluating models, especially when datasets are small.
How deep it goes
Splits data into k portions, trains k times with one portion held out each time, and averages validation scores.
Examples
Illustrates fourfold cross-validation by rotating the held-out validation portion across four training runs.
What you can do after
Evaluate a model by averaging validation scores from runs that each hold out a different data portion.

Read how Grokking Machine Learning explains cross-validation →

Financial AI in Practice

Cross-validation is a model evaluation technique that partitions data into subsets and repeatedly trains and evaluates a model on different subsets.

How it is used
Estimates credit-scoring performance across folds while training an XGBoost classifier on WoE-coded features.
How deep it goes
Implements stratified fold training and out-of-fold predictions, then recommends hold-out or champion–challenger checks against overfitting.
Examples
Uses five folds for WoE-coded credit data, with three or ten folds as alternatives and default/nondefault AUC checks.
What you can do after
Evaluate an XGBoost credit model with out-of-fold predictions and inspect fold-level feature importances.

Read how Financial AI in Practice explains cross-validation →

Also covered in

What to know first

Related concepts

Questions about cross-validation

What is cross-validation?

Cross-validation is a model-evaluation technique that repeatedly partitions available data into training and held-out subsets to estimate performance on unseen data.

Which Manning books explain cross-validation?

Ensemble Methods for Machine Learning, Causal Inference for Data Science, Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI, Build AI Drug Discovery Pipelines, Effective Conversational AI, Deep Learning with R, Third Edition, Timeless Algorithms, The Foundational Ideas, Deep Learning with Python, Third Edition and 7 more.

What should I know before learning cross-validation?

Manning books that teach cross-validation usually introduce language model, overfitting, machine learning and recall first.

What is the difference between cross-validation and out-of-bag evaluation?

Cross-validation is a model-evaluation technique that repeatedly partitions available data into training and held-out subsets to estimate performance on unseen data. Out-of-bag evaluation estimates a model’s predictive performance using observations omitted from the samples used to train the models making those predictions.

What are the parts of cross-validation?

Cross-validation includes cross-validation fold.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.