How deeply does Causal Inference for Data Science cover cross-validation?

Explains predictive-model selection and its causal limitations, but does not detail fold construction or scoring procedures.

Cross-validation

As explained in Causal Inference for Data Science

Cross-validation is a model-evaluation procedure that repeatedly partitions data into training and validation portions to estimate predictive performance.

Cross-validation estimates how well a trained predictive model is likely to perform on new data by repeatedly separating available data into training and validation portions. Its reassurance depends on a key condition: future data must behave similarly to past data. It answers whether a prediction system is likely to work as expected, not whether changing the system’s inputs will cause a desired outcome. section 1.2

Use cross-validation to compare alternative supervised-learning models and select the one with stronger predictive capability on held-out test data. For a classifier that supplies probabilities, such as a propensity-score model, evaluate discrimination with AUC rather than accuracy when the target is the probability of treatment. section 5.3.2 section 5.3.2

Accurate predictions can still mislead interventions

A game-level dataset makes bonuses decrease as ghosts increase, while both affect completion time. A regression using bonuses alone has an R-squared of 0.9969; adding ghosts raises it only to 0.9996, so both appear similarly predictive. But when a designer changes the historical relationship and adds ghosts and bonuses together, the models make sharply different predictions. Cross-validation could assess predictive error under historical-like data, yet it cannot establish which model correctly predicts the consequence of a new design. section 4.1.5

Keep the boundary between prediction and causal inference explicit:

Treat a good cross-validation result as evidence for prediction when the deployment setting resembles the observed data-generating setting. section 1.2
Do not select a causal model solely by cross-validation, because predictive fit does not determine the effect of an intervention. section 4.6
When machine learning supplies conditional probabilities for adjustment with many confounders, use cross-fitting even if cross-validation selected the predictive model. section 4.6
Interpret high AUC differently for propensity scores: near-perfect treatment prediction can indicate poor comparability between treated and untreated cases. section 5.3.2

How Causal Inference for Data Science covers it

Cover of Causal Inference for Data Science

liveBook manning.com

Read Causal Inference for Data Science on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Ensemble Methods for Machine LearningK-fold cross-validation partitions data into k folds and repeatedly trains and evaluates a model using one fold for validation and the others for training.
liveBook manning.com

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AICross-validation is a model-evaluation technique that repeatedly partitions data into training and validation subsets.
liveBook manning.com

Effective Conversational AIK-fold cross-validation is a model-evaluation technique that partitions a dataset into k folds and repeatedly evaluates a model with each fold held out in turn.
liveBook manning.com

Deep Learning with R, Third EditionK-fold cross-validation is an evaluation procedure that partitions data into K subsets and repeatedly trains on K−1 subsets while evaluating on the remaining subset.
liveBook manning.com

Deep Learning with Python, Third EditionA validation procedure that partitions data into K folds, trains on K−1 folds, and evaluates on the remaining fold in rotation.
liveBook manning.com

Grokking StatisticsCross-validation repeatedly partitions data into complementary training and evaluation folds to assess model performance.
liveBook manning.com

Delivering Machine Learning Projects: From design to deploymentCross-validation is a resampling technique that repeatedly trains and evaluates a model on different partitions of available data.
liveBook manning.com

Causal AICross-validation is a model evaluation technique that assesses predictive performance on data withheld from training.
liveBook manning.com

Building LLM Applications with DSPyCross-validation is an evaluation technique that repeatedly partitions data into training and validation subsets to produce a more reliable estimate.
liveBook manning.com

Grokking Machine LearningA resampling technique that partitions a dataset into k portions and repeatedly trains on k−1 portions while validating on the remaining portion, then averages the validation scores.
liveBook manning.com

Financial AI in PracticeCross-validation is a model evaluation technique that partitions data into subsets and repeatedly trains and evaluates a model on different subsets.

Compare how 12 books explain cross-validation →

Related concepts

covariateuses · A covariate is an observed or supplied variable included in a statistical, causal, or predictive model to describe, explain, predict, or adjust for variation in an outcome.
grid searchused by · Grid search is a search method that evaluates configurations formed from specified candidate values, typically every combination in their Cartesian product.
training datasetuses · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.

This explainer was generated by Manning's knowledge graph from the text of Causal Inference for Data Science. The section links point to the parts of the book it draws on.