liveBook · Manning

Causal inference

As explained in AI Model Evaluation

Causal analysis estimates the effects of actions or interventions on outcomes while accounting for how observations were generated.

Causal inference estimates what caused an outcome rather than treating correlation as proof. It helps determine whether an intervention—such as a recommendation, search ranking, or product change—actually produced a user or business effect. section 5.1

The central question is counterfactual: what would have happened if a different decision had been made? Engineering systems cannot replay that alternate reality directly, but logged decisions and suitable methods can estimate it before exposing a change to users. section 5.1 section 5.3.1

A practical causal-analysis workflow connects decisions, outcomes, and evidence:

  • Preserve the decision context, candidate actions, selected action, action probability or score, policy version, and observed outcome. section 5.11
  • Estimate the value of a different policy using off-policy evaluation; inverse propensity weighting reweights observed outcomes according to how likely the new policy was to select each logged action. section 5.11
  • Check overlap, propensity accuracy, weight stability, effective sample size, confidence intervals, and segment-level results before trusting the estimate. section 5.11 section 5.3

Recommendation policy comparison

An e-commerce site logs each user's context, the product shown, the serving policy's selection probability, and whether the user clicked or purchased. Those records can estimate how another recommendation policy might perform before an online test. The estimate remains conditional: if the new policy rarely chooses actions taken by the logged policy, the available evidence may be too weak for a reliable comparison. section 5.3 section 5.11

How AI Model Evaluation covers it

How deep it goes
Explains counterfactual reasoning, logging, off-policy estimation, and validation checks, but treats estimates as assumption-dependent rather than perfect replays.
Examples
Uses movie recommendations, e-commerce product recommendations, clicks, purchases, and alternative ranking or recommendation policies.
What it assumes
Assumes decision and outcome logging, while teaching why correlation alone cannot establish an intervention's effect.
What you can do after
Evaluate alternative recommendation or ranking policies before online exposure using logged decisions, outcomes, and policy probabilities.
How it connects
Connects causal inference with counterfactual evaluation, counterfactual logging, policy value, off-policy evaluation, and inverse propensity weighting.

Also explained in

Compare how 3 books explain causal inference →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.