Predictive error

As explained in Causal Inference for Data Science

Predictive error is the discrepancy between a model’s predicted outcome and the corresponding observed outcome.

Predictive error measures the discrepancy between a model’s predicted outcome and the corresponding observed outcome. Lower error can indicate useful prediction, but it does not guarantee that a model supports sound intervention decisions. section 4.1.5

Pac-Man level design

Two models predict level time from synthetic data: one uses bonuses, while the other uses bonuses and ghosts. Their R-squared values are 0.9969 and 0.9996, suggesting similarly low predictive error. Yet when a designer jointly changes ghosts and bonuses, the models predict 11.52 and 17.03 minutes. Predictive accuracy on historical designs therefore does not establish which model answers a design intervention. section 4.1.5

Figure 4.4 Number of ghosts versus number of bonuses. There is a strong correlation between them because they have a variable in common: the designer’s creative process. Points have been slightly perturbed to avoid point overlap.

Use the correlation plot to inspect how the designer’s creative process links ghosts and bonuses, creating a misleadingly strong predictive relationship. section 4.1.5

How Causal Inference for Data Science covers it

How deep it goes

Contrasts R-squared, cross-validation expectations, and intervention predictions, but does not develop a general error formula.

Examples

Uses synthetic Pac-Man levels, correlated ghosts and bonuses, two regression models, and competing intervention predictions.

What it assumes

Assumes familiarity with linear regression and R-squared while illustrating the distinction between prediction and causal reasoning.

What you can do after

Evaluate whether low predictive error remains relevant when planned changes break historical data relationships.

How it connects

Links predictive error to linear regression, R-squared, cross-validation, correlation, and causal inference.

liveBookmanning.com

Read Causal Inference for Data Science on liveBook Get the book at manning.com

Related concepts

• accuracycontrasts with · Accuracy is the degree to which an output, prediction, or measurement agrees with a correct or reference outcome under a specified criterion.

This explainer was generated by Manning's knowledge graph from the text of Causal Inference for Data Science. The section links point to the parts of the book it draws on.