liveBook · Manning

Counterfactual reasoning

As explained in AI Model Evaluation

Counterfactual evaluation estimates outcomes under decisions or policies other than those that generated the observed data.

Counterfactual reasoning asks what might have happened under a decision or policy different from the one that produced the observed data. In a recommender, you may know whether a user clicked Movie A, but not what that same user would have done if shown Movie B. Counterfactual evaluation estimates such missing outcomes; it does not reveal a true alternate history. section 5.0 section 5.2

Using production logs to evaluate a candidate policy can help teams iterate without exposing users to a potentially worse experience. It is especially useful when an A/B test is expensive, impractical, or risky. The estimate still depends on the data and assumptions, so it should complement rather than replace other evaluation methods. section 5.2.3 section 2.1.4

A practical evaluation has three parts:

  • Capture the decision: log its context, candidate set, selected action, selection probability, policy version, and observed outcome. section 5.3
  • Estimate the candidate policy’s value from logged outcomes. Off-policy evaluation can use inverse propensity weighting to reweight observations according to how likely the candidate and logging policies are to select each logged action. section 5.11 section 5.2.1
  • Compare the estimated value with the baseline, then check whether the evidence supports the comparison. Accurate propensities, sufficient policy overlap, and relevant logged context are essential. section 5.2.1
Figure 5.2 illustrates a simplified counterfactual evaluation workflow. Logged interaction data is reweighted using an off-policy correction method, such as inverse propensity weighting, to estimate the value of a candidate policy. That estimate is then compared with a baseline policy. The workflow assumes correct logging, sufficient overlap between the logged and candidate policies, and enough contextual information to support meaningful comparison.

Follow the workflow from logged interactions through reweighting to a candidate-versus-baseline comparison. section 5.2.1

Evaluating an agent’s tool routing

An agent can route a request to FAQ retrieval, live lookup, human handoff, or a direct response. Log the request and session context, candidate tools and their probabilities, the selected tool, and outcomes such as resolution or escalation. Calculate the new routing policy’s probabilities for those candidates and use inverse propensity weighting to estimate its resolution rate or escalation rate. section 5.9.3

More rows do not guarantee a reliable estimate. If the logging policy rarely explored actions the candidate policy would choose, only a small fraction of the data may provide useful evidence. Inspect effective sample size, confidence intervals, and weight distributions; test whether results change substantially when large weights are clipped or the time window changes. section 5.3 section 5.3.2

Design logging around the decision and outcome you intend to evaluate. In large action spaces, preserve the candidate set actually considered rather than trying to log every possible action. Balance the detail needed for evaluation against latency, storage, privacy, and maintenance costs. section 5.5 section 5.5.1 section 5.11

How AI Model Evaluation covers it

How deep it goes
Explains decision logging, policy-value estimation, assumptions, and stability checks without treating estimates as exact alternate histories.
Examples
Uses movie recommendations and an agent routing requests among four tools to illustrate alternative decisions and outcomes.
What you can do after
Evaluate alternative policies using logged decisions, propensity weighting, baseline comparisons, and reliability checks.
How it connects
Connects causal inference, off-policy evaluation, inverse propensity weighting, policy overlap, and A/B testing.

Also explained in

Compare how 2 books explain counterfactual reasoning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.