Counterfactual reasoning asks what might have happened under a decision or policy different from the one that produced the observed data. In a recommender, you may know whether a user clicked Movie A, but not what that same user would have done if shown Movie B. Counterfactual evaluation estimates such missing outcomes; it does not reveal a true alternate history. section 5.0 section 5.2
Using production logs to evaluate a candidate policy can help teams iterate without exposing users to a potentially worse experience. It is especially useful when an A/B test is expensive, impractical, or risky. The estimate still depends on the data and assumptions, so it should complement rather than replace other evaluation methods. section 5.2.3 section 2.1.4
A practical evaluation has three parts:
- Capture the decision: log its context, candidate set, selected action, selection probability, policy version, and observed outcome. section 5.3
- Estimate the candidate policy’s value from logged outcomes. Off-policy evaluation can use inverse propensity weighting to reweight observations according to how likely the candidate and logging policies are to select each logged action. section 5.11 section 5.2.1
- Compare the estimated value with the baseline, then check whether the evidence supports the comparison. Accurate propensities, sufficient policy overlap, and relevant logged context are essential. section 5.2.1
Figure 5.2 illustrates a simplified counterfactual evaluation workflow. Logged interaction data is reweighted using an off-policy correction method, such as inverse propensity weighting, to estimate the value of a candidate policy. That estimate is then compared with a baseline policy. The workflow assumes correct logging, sufficient overlap between the logged and candidate policies, and enough contextual information to support meaningful comparison.
Follow the workflow from logged interactions through reweighting to a candidate-versus-baseline comparison. section 5.2.1
Evaluating an agent’s tool routing
An agent can route a request to FAQ retrieval, live lookup, human handoff, or a direct response. Log the request and session context, candidate tools and their probabilities, the selected tool, and outcomes such as resolution or escalation. Calculate the new routing policy’s probabilities for those candidates and use inverse propensity weighting to estimate its resolution rate or escalation rate. section 5.9.3
More rows do not guarantee a reliable estimate. If the logging policy rarely explored actions the candidate policy would choose, only a small fraction of the data may provide useful evidence. Inspect effective sample size, confidence intervals, and weight distributions; test whether results change substantially when large weights are clipped or the time window changes. section 5.3 section 5.3.2
Design logging around the decision and outcome you intend to evaluate. In large action spaces, preserve the candidate set actually considered rather than trying to log every possible action. Balance the detail needed for evaluation against latency, storage, privacy, and maintenance costs. section 5.5 section 5.5.1 section 5.11

