Cross-validation

As explained in Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

Cross-validation is a model-evaluation technique that repeatedly partitions data into training and validation subsets.

Cross-validation estimates how a machine-learning model will perform on future data drawn from the same distribution as the training data. Rather than trusting one train/validation split, it repeatedly trains on subsets of the available data and evaluates on the held-out portion. This makes it a standard basis for comparing tabular-data models before fitting a selected approach on all available data. section 4.1 section 4.3

In K-fold cross-validation, data is divided into K folds. Each fold serves as validation once while the remaining folds train the model, so every observation is evaluated as held-out data. Fold-level metrics are then aggregated, commonly with a mean or median; reporting the average and standard deviation helps compare both expected error and performance consistency across samples. section 4.3 section 7.2.3

Figure 4.9 How data is distributed between train and validation across the folds of a five-fold cross-validation strategy

A five-fold layout makes the rotation concrete: every fold takes a turn as validation while the others provide training data. section 4.3

Build a useful cross-validation workflow by:

• Putting feature transformations and the model in one pipeline, so they are evaluated together. section 4.1 section 4.3
• Selecting an evaluation measure suited to the task, such as RMSE for regression or accuracy for balanced classification. section 4.3
• Using the same validation strategy and metrics when comparing candidate models. section 4.3
• Refitting the accepted pipeline on all data after cross-validation results are suitable. section 4.1

The split strategy must reflect the data. Ordinary K-fold works well in many cases, but stratified K-fold preserves the distribution of a chosen feature in every fold. For geographically influenced real-estate values, preserving a similar territory mix avoids folds that poorly represent the full dataset. Small groups may need preparation before splitting; otherwise, they cannot be represented reliably across folds. section 7.2.1

Diagnosing a real-estate baseline

Cross-validation can produce held-out predictions for every training case, not only a summary score. A linear-regression baseline for accommodation prices uses validation-fold predictions to calculate R-squared, RMSE, and MAE, inspect implausible negative predictions, and compare predictions with actual targets. This exposes underfitting or overfitting patterns across subsets of the data. section 7.2.3

Cross-validation also supports hyperparameter optimization. Define a search space, choose evaluation metrics, and use an appropriate validation scheme so settings are selected for generalization rather than merely for favorable results on the available data. For tabular deep learning in Keras, repeated randomized training runs with train, validation, and test proportions are presented as providing these benefits naturally. section 6.3 section 9.4.1

How Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI covers it

How deep it goes

Explains repeated holdouts, K-fold aggregation, stratification, prediction-based diagnostics, and tuning-oriented validation schemes.

Examples

Uses synthetic logistic classification, Airbnb and real-estate data, stratified neighborhood folds, and linear-regression prediction diagnostics.

What it assumes

Assumes a prepared tabular dataset, model pipeline, and task-appropriate evaluation metric.

What you can do after

Apply it via Scikit-learn to evaluate pipelines, compare models, inspect held-out predictions, and select validation-aware hyperparameters.

How it connects

Connects pipelines, RMSE, accuracy, hyperparameter optimization, stratified sampling, baseline models, and Keras validation.

liveBookmanning.com

Read Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningK-fold cross-validation partitions data into k folds and repeatedly trains and evaluates a model using one fold for validation and the others for training.
• 

  liveBookmanning.com
  Causal Inference for Data ScienceCross-validation is a model-evaluation procedure that repeatedly partitions data into training and validation portions to estimate predictive performance.
• 

  liveBookmanning.com
  Effective Conversational AIK-fold cross-validation is a model-evaluation technique that partitions a dataset into k folds and repeatedly evaluates a model with each fold held out in turn.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionK-fold cross-validation is an evaluation procedure that partitions data into K subsets and repeatedly trains on K−1 subsets while evaluating on the remaining subset.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA validation procedure that partitions data into K folds, trains on K−1 folds, and evaluates on the remaining fold in rotation.
• 

  liveBookmanning.com
  Grokking StatisticsCross-validation repeatedly partitions data into complementary training and evaluation folds to assess model performance.
• 

  liveBookmanning.com
  Delivering Machine Learning Projects: From design to deploymentCross-validation is a resampling technique that repeatedly trains and evaluates a model on different partitions of available data.
• 

  liveBookmanning.com
  Causal AICross-validation is a model evaluation technique that assesses predictive performance on data withheld from training.
• 

  liveBookmanning.com
  Building LLM Applications with DSPyCross-validation is an evaluation technique that repeatedly partitions data into training and validation subsets to produce a more reliable estimate.
• 

  liveBookmanning.com
  Grokking Machine LearningA resampling technique that partitions a dataset into k portions and repeatedly trains on k−1 portions while validating on the remaining portion, then averages the validation scores.
• 

  liveBookmanning.com
  Financial AI in PracticeCross-validation is a model evaluation technique that partitions data into subsets and repeatedly trains and evaluates a model on different subsets.

Compare how 12 books explain cross-validation →

Related concepts

• covariateuses · A covariate is an observed or supplied variable included in a statistical, causal, or predictive model to describe, explain, predict, or adjust for variation in an outcome.
• grid searchused by · Grid search is a search method that evaluates configurations formed from specified candidate values, typically every combination in their Cartesian product.
• training datasetuses · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI. The section links point to the parts of the book it draws on.