Cross-fitting

As explained in Causal Inference for Data Science

Cross-fitting is a procedure that partitions data, trains models on complementary subsets, and evaluates predictions on held-out subsets to reduce overfitting in causal estimation.

Cross-fitting separates model training from the predictions used for causal estimation. Rather than letting a model predict the same observations it learned from, split the data: fit on one portion and predict on the held-out portion. This guards against a flexible model fitting noise in the historical data and carrying that overfitting into an estimated causal effect. section 4.2.5 section 4.6

In two-fold cross-fitting, divide the data into D1 and D2. Train on D1 and make the needed counterfactual or nuisance predictions for D2; then reverse the roles. Combine the two held-out results, such as by averaging fold-specific ATE estimates. For k-fold cross-fitting, train on the complement of each fold, predict within that fold, and average across folds. section 4.2.5 section 8.1.3

Table 4.5 The difference between training and predicting on the same dataset with respect to applying cross-fitting

Training + predicting	Cross-fitting
Training and predicting on the same dataset can lead to overfitting, even when using cross-validation.	Two-fold cross-fitting trains on one dataset but predicts on the other. Cross-fitting avoids overfitting on the same dataset.

The key distinction is not merely tuning a model well: causal predictions must be made on observations that were excluded from that model’s training data. section 4.2.5

A practical two-fold workflow is:

• Split observations into two similarly sized folds; when treatment is binary, keep treatment proportions balanced across folds. section 4.2.5
• Within each training fold, tune and fit the predictive model or models required by the causal estimator. section 5.3.2 section 5.4.1
• Generate predictions only for the opposite held-out fold, then switch fold roles. section 4.2.5
• Compute the causal quantity from held-out predictions and combine results across folds. section 4.2.5 section 8.1.3

Propensity scores for patient data

To estimate propensity scores, train a treatment-prediction model on D1 and score patients in D2; train another model on D2 and score patients in D1. Each patient’s propensity score therefore comes from a model that did not train on that patient. Compare this result with same-data predictions, whose AUC can look better than the cross-validated AUC because of overfitting. section 5.3.2 section 5.4.1

Cross-fitting complements, rather than replaces, causal design. Cross-validation can select predictive hyperparameters, but it does not by itself prevent same-data causal predictions. Cross-fitting also does not resolve confounding: machine-learning adjustment still depends on including the relevant confounders. section 4.6 section 8.1.3

The same split-and-switch pattern works with S-learners, T-learners, propensity-score adjustment, and Double Machine Learning. In DoubleML, k-fold cross-fitting can be requested with n_folds = k; tuning strategy still matters because tuning before the fold procedure can be faster but potentially biased. section 4.2.5 section 8.1.3

How Causal Inference for Data Science covers it

How deep it goes

Details two-fold and k-fold workflows, including held-out prediction and fold aggregation, without deriving formal guarantees.

Examples

Uses held-out propensity-score prediction and RHC treatment-effect exercises, plus S-learner, T-learner, and DML folds.

What it assumes

Uses historical treatment, outcome, and confounder data; leaves valid confounder selection to domain expertise.

What you can do after

Apply it to produce held-out propensity-score and treatment-effect predictions, then aggregate fold-specific causal estimates.

How it connects

Connects adjustment formula, S-learner, T-learner, propensity scores, cross-validation, and Double Machine Learning.

liveBookmanning.com

Read Causal Inference for Data Science on liveBook Get the book at manning.com

Related concepts

• overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• propensity scoreuses · A propensity score is the estimated conditional probability that a unit receives a particular treatment or action given observed covariates or context.
• T-learnerused by · A T-learner is a causal inference method that fits separate outcome models for each treatment level and contrasts their predictions.
• double machine learninguses · Double machine learning is a causal-effect estimation method that uses machine-learning nuisance models with orthogonalized estimation to reduce bias from nuisance-function errors.
• k-nearest neighborsconfigures · K-nearest neighbors is a method that predicts a query’s class or value from the labels or values of its k closest training examples, as measured by a specified distance or similarity.

This explainer was generated by Manning's knowledge graph from the text of Causal Inference for Data Science. The section links point to the parts of the book it draws on.