liveBook · Manning

Accuracy

As explained in Causal Inference for Data Science

Accuracy is the proportion of predictions that match the corresponding observed outcomes.

Accuracy is the proportion of predictions that match observed outcomes. It answers how well a model predicts, not whether an action changes what happens. section 3.3.1

A perfectly accurate recommendation

Suppose a customer buys bread every Monday, Wednesday, and Friday. A recommender can predict the next purchase with 100% accuracy, yet the customer would buy bread without seeing the recommendation. The prediction is accurate, while the recommender’s causal effect on buying is zero. section 3.3.1

Interpret accuracy according to the question being asked:

  • Use predictive accuracy to assess how well a model forecasts outcomes on new data. section 5.3.2
  • Do not treat high accuracy as evidence that a recommendation or treatment caused the outcome; compare treated and untreated groups when causal efficacy matters. section 3.3.1
  • For propensity scores, prefer AUC to accuracy because perfect treatment prediction can violate the positivity assumption; lower AUC may be better. section 5.3.2
Figure 3.12 Graph of a pure recommender system that does not influence the customer. This system may predict customer behavior accurately, but it does not affect their actions.
figure

Use the causal graph to separate accurate prediction from an arrow representing influence on customer behavior. section 3.3.1

How Causal Inference for Data Science covers it

How deep it goes
Defines accuracy and illustrates its causal limitation, then defers propensity-score evaluation to AUC.
Examples
Uses routine bread purchases, a perfectly accurate recommender, and treatment-assignment classifiers.
What you can do after
Evaluate predictive performance and distinguish accurate forecasts from evidence of causal impact.
How it connects
Connects predictive modeling with recommender systems, causal inference, A/B testing, propensity scores, AUC, and positivity.
Cover of Causal Inference for Data Science
liveBookmanning.com
Read Causal Inference for Data Science on liveBook Get the book at manning.com

Also explained in

Compare how 32 books explain accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Causal Inference for Data Science. The section links point to the parts of the book it draws on.