Cross-validation
Cross-validation is a model-evaluation technique that repeatedly partitions available data into training and held-out subsets to estimate performance on unseen data.
Ensemble Methods for Machine Learning
K-fold cross-validation partitions data into k folds and repeatedly trains and evaluates a model using one fold for validation and the others for training.
- How it is used
- Evaluates models, simulates out-of-sample testing, and selects hyperparameters such as learning rate.
- How deep it goes
- Explains fold construction, repeated training and validation, stratification, averaging, refitting, and out-of-fold stacking features.
- Examples
- Uses five-fold stacking, breast-cancer learning-rate searches, digit-classification grid search, and CatBoost credit-approval tuning.
- What you can do after
- Apply it via scikit-learn, LightGBM, XGBoost, or CatBoost to compare settings and refit selected models.
Read how Ensemble Methods for Machine Learning explains cross-validation →
Causal Inference for Data Science
Cross-validation is a model-evaluation procedure that repeatedly partitions data into training and validation portions to estimate predictive performance.
- How it is used
- Evaluates and selects supervised predictive models before they are used on future, historically similar data.
- How deep it goes
- Explains predictive-model selection and its causal limitations, but does not detail fold construction or scoring procedures.
- Examples
- Uses house inspection versus earthquake resistance and Pac-Man level regressions with bonuses, ghosts, and completion time.
- What you can do after
- Evaluate predictive models by held-out capability while avoiding cross-validation alone as a basis for causal-model choice.
Read how Causal Inference for Data Science explains cross-validation →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Cross-validation is a model-evaluation technique that repeatedly partitions data into training and validation subsets.
- How it is used
- Estimates expected performance on future tabular data and supports model comparison before final fitting.
- How deep it goes
- Explains repeated holdouts, K-fold aggregation, stratification, prediction-based diagnostics, and tuning-oriented validation schemes.
- Examples
- Uses synthetic logistic classification, Airbnb and real-estate data, stratified neighborhood folds, and linear-regression prediction diagnostics.
- What you can do after
- Apply it via Scikit-learn to evaluate pipelines, compare models, inspect held-out predictions, and select validation-aware hyperparameters.
Build AI Drug Discovery Pipelines
Cross-validation estimates model performance or selects configurations by repeatedly training and evaluating on different partitions of the training data.
- How it is used
- Uses training data efficiently to compare models, tune configurations, and estimate generalization before final test evaluation.
- How deep it goes
- Explains fold construction, averaging, metrics, leakage, and retraining, but defers production deployment and maintenance.
- Examples
- Demonstrates Scikit-Learn fingerprints, five-fold accuracy, a dummy-classifier baseline, stratified molecular classification, and graph-network validation.
- What you can do after
- Apply it via Scikit-Learn to compare models, tune configurations, calculate metrics, and select a candidate without opening the test set.
Read how Build AI Drug Discovery Pipelines explains cross-validation →
Effective Conversational AI
K-fold cross-validation is a model-evaluation technique that partitions a dataset into k folds and repeatedly evaluates a model with each fold held out in turn.
- How it is used
- Measures a chatbot intent classifier's baseline performance when additional annotated test data is unavailable.
- How deep it goes
- Outlines repeated holdout, scoring, and reintegration, but leaves detailed testing instructions to further guidance.
- Examples
- Uses a 125-question chatbot baseline with 84% accuracy to show how sparse intents can produce zero recall.
- What you can do after
- Evaluate baseline classifier performance and identify weak intents while treating results cautiously.
Read how Effective Conversational AI explains cross-validation →
Deep Learning with R, Third Edition
K-fold cross-validation is an evaluation procedure that partitions data into K subsets and repeatedly trains on K−1 subsets while evaluating on the remaining subset.
- How it is used
- Guides model evaluation and training-duration choices when hold-out validation is too variable because little data is available.
- How deep it goes
- Explains partitioning, repeated training, score averaging, validation histories, plotting, and final-model training without formal derivation.
- Examples
- Demonstrates three- and four-fold splits, validation MAE histories, score plots, and a final regression model.
- What you can do after
- Apply it via an R training loop to estimate validation performance, compare epochs, and select a final training duration.
Read how Deep Learning with R, Third Edition explains cross-validation →
Timeless Algorithms, The Foundational Ideas
Cross-validation is a model evaluation technique that repeatedly withholds portions of data from fitting and assesses performance on those held-out observations.
- How it is used
- Selects a soft-margin SVM penalty value by estimating how candidate models perform on withheld observations.
- How deep it goes
- Defines k-fold evaluation through model fitting, held-out scoring, fold rotation, and averaging, while leaving k to practical judgment.
- Examples
- Compares five-fold and ten-fold choices and names accuracy, balanced accuracy, precision, recall, and F1 as evaluation measures.
- What you can do after
- Apply fold-based held-out evaluation to compare candidate SVM penalty values by average validation performance.
Read how Timeless Algorithms, The Foundational Ideas explains cross-validation →
Deep Learning with Python, Third Edition
A validation procedure that partitions data into K folds, trains on K−1 folds, and evaluates on the remaining fold in rotation.
- How it is used
- Stabilizes validation-based model comparison when limited data makes a single hold-out score unreliable.
- How deep it goes
- Implements fold construction, repeated training, score averaging, per-epoch histories, plotting, and practical epoch selection.
- Examples
- Uses four-fold scalar regression, validation MAE histories, epoch plots, and a final 130-epoch model.
- What you can do after
- Apply it via a model-training loop to obtain a more reliable validation estimate and guide training duration.
Read how Deep Learning with Python, Third Edition explains cross-validation →
Vibe Engineering
Cross-validation is a model-evaluation technique that repeatedly trains and evaluates a model on different partitions of a dataset.
- How it is used
- Checks whether unusually high model scores hold across data partitions when overfitting is a concern.
- How deep it goes
- Explains five-fold training and testing and reports fold accuracies, while linking to the implementation.
- Examples
- Uses a medicine-risk model with near-perfect precision–recall curves and reports five fold accuracies.
- What you can do after
- Evaluate and understand fold-to-fold accuracy variation as a check on model performance.
Grokking Statistics
Cross-validation repeatedly partitions data into complementary training and evaluation folds to assess model performance.
- How it is used
- Uses withheld data to check whether classification models generalize to unseen cases.
- How deep it goes
- Demonstrates three-fold iteration and prediction, but stops short of detailed statistical theory or advanced validation metrics.
- Examples
- Uses a toddler-meltdown classification dataset and a three-fold KFold implementation with scikit-learn.
- What you can do after
- Apply it via scikit-learn’s KFold to evaluate logistic-regression predictions on repeatedly held-out data.
Delivering Machine Learning Projects: From design to deployment
Cross-validation is a resampling technique that repeatedly trains and evaluates a model on different partitions of available data.
- How it is used
- Supports model testing and selection when data is scarce or a simple train/test split would waste available training data.
- How deep it goes
- Defines rotating train/test partitions, score aggregation, leave-one-out evaluation, and computational limits without developing algorithmic internals.
- Examples
- Uses ten-fold evaluation, an eight-partition P1–P8 matrix, and leave-one-out evaluation across 1,000 examples.
- What you can do after
- Evaluate model performance by rotating held-out partitions, aggregating scores, and documenting measurement trade-offs for selection.
Read how Delivering Machine Learning Projects: From design to deployment explains cross-validation →
Causal AI
Cross-validation is a model evaluation technique that assesses predictive performance on data withheld from training.
- How it is used
- Compares models by estimating predictive performance on data excluded from training.
- How deep it goes
- Defines held-out prediction and contrasts it with goodness-of-fit, but does not specify fold construction or computational procedure.
- Examples
- Uses a model with strong training-data fit but poor withheld-data prediction as the central comparison.
- What you can do after
- Evaluate models by distinguishing generalization to unseen data from fit to training data.
Building LLM Applications with DSPy
Cross-validation is an evaluation technique that repeatedly partitions data into training and validation subsets to produce a more reliable estimate.
- How it is used
- Estimates model skill more reliably than a single train-validation-test split.
- How deep it goes
- Explains repeated training-validation splits, their reliability benefit, and their cost without detailing implementation.
- What you can do after
- Evaluate model skill using repeated splits and weigh greater reliability against added cost.
Read how Building LLM Applications with DSPy explains cross-validation →
Grokking Machine Learning
A resampling technique that partitions a dataset into k portions and repeatedly trains on k−1 portions while validating on the remaining portion, then averages the validation scores.
- How it is used
- Reuses data for training and validation when evaluating models, especially when datasets are small.
- How deep it goes
- Splits data into k portions, trains k times with one portion held out each time, and averages validation scores.
- Examples
- Illustrates fourfold cross-validation by rotating the held-out validation portion across four training runs.
- What you can do after
- Evaluate a model by averaging validation scores from runs that each hold out a different data portion.
Read how Grokking Machine Learning explains cross-validation →
Financial AI in Practice
Cross-validation is a model evaluation technique that partitions data into subsets and repeatedly trains and evaluates a model on different subsets.
- How it is used
- Estimates credit-scoring performance across folds while training an XGBoost classifier on WoE-coded features.
- How deep it goes
- Implements stratified fold training and out-of-fold predictions, then recommends hold-out or champion–challenger checks against overfitting.
- Examples
- Uses five folds for WoE-coded credit data, with three or ten folds as alternatives and default/nondefault AUC checks.
- What you can do after
- Evaluate an XGBoost credit model with out-of-fold predictions and inspect fold-level feature importances.
Read how Financial AI in Practice explains cross-validation →
Also covered in
- Time Series Forecasting Using Foundation Models
- Machine Learning System Design
- AI-Powered Search
- Fight Fraud with Machine LearningIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- overfittingOverfitting is a modeling condition in which a model adapts too closely to particular data, including its noise or idiosyncrasies, and consequently generalizes poorly to new data.
- machine learningMachine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
- recallRecall is the proportion of all relevant or actual positive items that a system successfully retrieves or correctly identifies.
Related concepts
- RandomizedSearchCVused by · RandomizedSearchCV is a scikit-learn estimator that samples hyperparameter configurations from specified distributions or candidate lists and evaluates them using cross-validation.
- covariateuses · A covariate is an observed or supplied variable included in a statistical, causal, or predictive model to describe, explain, predict, or adjust for variation in an outcome.
- out-of-bag evaluationcontrasts with · Out-of-bag evaluation estimates a model’s predictive performance using observations omitted from the samples used to train the models making those predictions.
- cross-validation foldhas part · A cross-validation fold is one of the subsets into which data is partitioned for training or evaluation in a cross-validation procedure.
- grid searchused by · Grid search is a search method that evaluates configurations formed from specified candidate values, typically every combination, to identify a high-performing configuration.
- training datasetuses · A training dataset is a collection of examples supplied to a learning procedure to fit or adapt a model’s parameters or behavior.
Questions about cross-validation
What is cross-validation?
Cross-validation is a model-evaluation technique that repeatedly partitions available data into training and held-out subsets to estimate performance on unseen data.
Which Manning books explain cross-validation?
Ensemble Methods for Machine Learning, Causal Inference for Data Science, Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI, Build AI Drug Discovery Pipelines, Effective Conversational AI, Deep Learning with R, Third Edition, Timeless Algorithms, The Foundational Ideas, Deep Learning with Python, Third Edition and 7 more.
What should I know before learning cross-validation?
Manning books that teach cross-validation usually introduce language model, overfitting, machine learning and recall first.
What is the difference between cross-validation and out-of-bag evaluation?
Cross-validation is a model-evaluation technique that repeatedly partitions available data into training and held-out subsets to estimate performance on unseen data. Out-of-bag evaluation estimates a model’s predictive performance using observations omitted from the samples used to train the models making those predictions.
What are the parts of cross-validation?
Cross-validation includes cross-validation fold.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.














