How deeply does Causal AI cover goodness-of-fit measure?

Defines fit against training data and contrasts it with held-out prediction, but stops short of identifying the true data-generating process.

Goodness-of-fit measure

As explained in Causal AI

A goodness-of-fit statistic quantifies how well a model fits or accounts for observed data.

A model can fit its training data well without predicting new data well or representing the true causal process. A goodness-of-fit statistic measures how closely a model fits the data used to train it. section 2.3.4

Use goodness-of-fit measures with these distinctions in mind:

Separate goodness-of-fit from cross-validation: fit statistics evaluate training data, whereas cross-validation generally evaluates prediction on data not used for training. section 2.3.4
Do not treat a strong fit as causal evidence; multiple data-generating processes can explain the same dataset, and fit alone cannot identify the true one. section 2.3.4
In score-based causal discovery, compare candidate directed acyclic graphs by their goodness-of-fit scores while incorporating available structural constraints or prior knowledge. section 4.6.1

Model selection versus causal discovery

A model may have better goodness-of-fit than alternatives yet predict poorly under cross-validation. Conversely, a good predictor may support bogus causal inferences. Score-based causal discovery instead searches for causal DAGs with high goodness-of-fit scores. section 2.3.4 section 4.6.1

How Causal AI covers it

liveBook manning.com

Read Causal AI on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Statistics Every Programmer NeedsGoodness of fit is the degree to which a statistical model's predictions agree with observed data.

Compare how 2 books explain goodness-of-fit measure →

Related concepts

coefficient of determinationmeasured by · The coefficient of determination, or R-squared, is a regression metric measuring the proportion of target variation explained by a fitted model relative to a mean-value baseline.
linear regressionmeasured by · Linear regression is a statistical modeling method that estimates a continuous response as a linear combination of predictor variables and fitted coefficients.
McFadden's pseudo R-squaredmeasured by · McFadden's pseudo R-squared is a likelihood-based goodness-of-fit measure comparing a fitted discrete-choice model with an intercept-only null model.
cross-validationcontrasts with · Cross-validation is a resampling technique that evaluates a model by repeatedly fitting it on some portions of available data and assessing it on held-out portions.
chi-square testmeasured by · A chi-square test is a statistical hypothesis test that uses a chi-square-distributed statistic to evaluate hypotheses about categorical data, goodness of fit, or variance.
AICmeasured by · The Akaike information criterion is a model-selection score that balances goodness of fit against model complexity, with lower values generally preferred among comparable models.

This explainer was generated by Manning's knowledge graph from the text of Causal AI. The section links point to the parts of the book it draws on.