Cross-validation
Cross-validation is a resampling technique that evaluates a model by repeatedly fitting it on some portions of available data and assessing it on held-out portions.
Ensemble Methods for Machine Learning
K-fold cross-validation partitions data into k folds and repeatedly trains and evaluates a model using one fold for validation and the others for training.
- How it is used
- Evaluates models, simulates out-of-sample testing, and selects hyperparameters such as learning rate.
- How deep it goes
- Explains fold construction, repeated training and validation, stratification, averaging, refitting, and out-of-fold stacking features.
- Examples
- Uses five-fold stacking, breast-cancer learning-rate searches, digit-classification grid search, and CatBoost credit-approval tuning.
- What you can do after
- Apply it via scikit-learn, LightGBM, XGBoost, or CatBoost to compare settings and refit selected models.
Read how Ensemble Methods for Machine Learning explains cross-validation →
Causal Inference for Data Science
Cross-validation is a model-evaluation procedure that repeatedly partitions data into training and validation portions to estimate predictive performance.
- How it is used
- Evaluates and selects supervised predictive models before they are used on future, historically similar data.
- How deep it goes
- Explains predictive-model selection and its causal limitations, but does not detail fold construction or scoring procedures.
- Examples
- Uses house inspection versus earthquake resistance and Pac-Man level regressions with bonuses, ghosts, and completion time.
- What you can do after
- Evaluate predictive models by held-out capability while avoiding cross-validation alone as a basis for causal-model choice.
Read how Causal Inference for Data Science explains cross-validation →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Cross-validation is a model-evaluation technique that repeatedly partitions data into training and validation subsets.
- How it is used
- Estimates expected performance on future tabular data and supports model comparison before final fitting.
- How deep it goes
- Explains repeated holdouts, K-fold aggregation, stratification, prediction-based diagnostics, and tuning-oriented validation schemes.
- Examples
- Uses synthetic logistic classification, Airbnb and real-estate data, stratified neighborhood folds, and linear-regression prediction diagnostics.
- What you can do after
- Apply it via Scikit-learn to evaluate pipelines, compare models, inspect held-out predictions, and select validation-aware hyperparameters.
Effective Conversational AI
K-fold cross-validation is a model-evaluation technique that partitions a dataset into k folds and repeatedly evaluates a model with each fold held out in turn.
- How it is used
- Measures a chatbot intent classifier's baseline performance when additional annotated test data is unavailable.
- How deep it goes
- Outlines repeated holdout, scoring, and reintegration, but leaves detailed testing instructions to further guidance.
- Examples
- Uses a 125-question chatbot baseline with 84% accuracy to show how sparse intents can produce zero recall.
- What you can do after
- Evaluate baseline classifier performance and identify weak intents while treating results cautiously.
Read how Effective Conversational AI explains cross-validation →
Deep Learning with R, Third Edition
K-fold cross-validation is an evaluation procedure that partitions data into K subsets and repeatedly trains on K−1 subsets while evaluating on the remaining subset.
- How it is used
- Guides model evaluation and training-duration choices when hold-out validation is too variable because little data is available.
- How deep it goes
- Explains partitioning, repeated training, score averaging, validation histories, plotting, and final-model training without formal derivation.
- Examples
- Demonstrates three- and four-fold splits, validation MAE histories, score plots, and a final regression model.
- What you can do after
- Apply it via an R training loop to estimate validation performance, compare epochs, and select a final training duration.
Read how Deep Learning with R, Third Edition explains cross-validation →
Deep Learning with Python, Third Edition
A validation procedure that partitions data into K folds, trains on K−1 folds, and evaluates on the remaining fold in rotation.
- How it is used
- Stabilizes validation-based model comparison when limited data makes a single hold-out score unreliable.
- How deep it goes
- Implements fold construction, repeated training, score averaging, per-epoch histories, plotting, and practical epoch selection.
- Examples
- Uses four-fold scalar regression, validation MAE histories, epoch plots, and a final 130-epoch model.
- What you can do after
- Apply it via a model-training loop to obtain a more reliable validation estimate and guide training duration.
Read how Deep Learning with Python, Third Edition explains cross-validation →
Grokking Statistics
Cross-validation repeatedly partitions data into complementary training and evaluation folds to assess model performance.
- How it is used
- Uses withheld data to check whether classification models generalize to unseen cases.
- How deep it goes
- Demonstrates three-fold iteration and prediction, but stops short of detailed statistical theory or advanced validation metrics.
- Examples
- Uses a toddler-meltdown classification dataset and a three-fold KFold implementation with scikit-learn.
- What you can do after
- Apply it via scikit-learn’s KFold to evaluate logistic-regression predictions on repeatedly held-out data.
Delivering Machine Learning Projects: From design to deployment
Cross-validation is a resampling technique that repeatedly trains and evaluates a model on different partitions of available data.
- How it is used
- Supports model testing and selection when data is scarce or a simple train/test split would waste available training data.
- How deep it goes
- Defines rotating train/test partitions, score aggregation, leave-one-out evaluation, and computational limits without developing algorithmic internals.
- Examples
- Uses ten-fold evaluation, an eight-partition P1–P8 matrix, and leave-one-out evaluation across 1,000 examples.
- What you can do after
- Evaluate model performance by rotating held-out partitions, aggregating scores, and documenting measurement trade-offs for selection.
Read how Delivering Machine Learning Projects: From design to deployment explains cross-validation →
Causal AI
Cross-validation is a model evaluation technique that assesses predictive performance on data withheld from training.
- How it is used
- Compares models by estimating predictive performance on data excluded from training.
- How deep it goes
- Defines held-out prediction and contrasts it with goodness-of-fit, but does not specify fold construction or computational procedure.
- Examples
- Uses a model with strong training-data fit but poor withheld-data prediction as the central comparison.
- What you can do after
- Evaluate models by distinguishing generalization to unseen data from fit to training data.
Building LLM Applications with DSPy
Cross-validation is an evaluation technique that repeatedly partitions data into training and validation subsets to produce a more reliable estimate.
- How it is used
- Estimates model skill more reliably than a single train-validation-test split.
- How deep it goes
- Explains repeated training-validation splits, their reliability benefit, and their cost without detailing implementation.
- What you can do after
- Evaluate model skill using repeated splits and weigh greater reliability against added cost.
Read how Building LLM Applications with DSPy explains cross-validation →
Grokking Machine Learning
A resampling technique that partitions a dataset into k portions and repeatedly trains on k−1 portions while validating on the remaining portion, then averages the validation scores.
- How it is used
- Reuses data for training and validation when evaluating models, especially when datasets are small.
- How deep it goes
- Splits data into k portions, trains k times with one portion held out each time, and averages validation scores.
- Examples
- Illustrates fourfold cross-validation by rotating the held-out validation portion across four training runs.
- What you can do after
- Evaluate a model by averaging validation scores from runs that each hold out a different data portion.
Read how Grokking Machine Learning explains cross-validation →
Financial AI in Practice
Cross-validation is a model evaluation technique that partitions data into subsets and repeatedly trains and evaluates a model on different subsets.
- How it is used
- Uses cross-validated performance to evaluate a credit-scoring model in an end-to-end BFSI workflow.
- How deep it goes
- Mentions cross-validated evaluation but does not specify fold setup or evaluation procedure.
- Examples
- Pairs cross-validated performance review with confusion matrices for a WoE-based credit model.
- What you can do after
- Evaluate credit-model performance as part of a credit-scoring workflow.
Read how Financial AI in Practice explains cross-validation →
Related concepts
- covariateuses · A covariate is an observed or supplied variable included in a statistical, causal, or predictive model to describe, explain, predict, or adjust for variation in an outcome.
- grid searchused by · Grid search is a search method that evaluates configurations formed from specified candidate values, typically every combination in their Cartesian product.
- training datasetuses · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.