liveBook · Manning

Chapter 5: Counterfactual evaluations

What you will learn in chapter 5 of AI Model Evaluation

Introduces counterfactual evaluations as estimates of how an AI system might have performed had it made different decisions.

After this chapter you will be able to

What it builds on

Section by section

Introduction

Introduces counterfactual evaluations as estimates of how an AI system might have performed had it made different decisions.

Key terms: counterfactual evaluations, logged production data, decision probabilities, observed outcomes, assumptions

5.1 What is causal inference

The section introduces causal inference as a way to distinguish cause from coincidence and estimate what might have happened under a different decision.

Key terms: causal inference, cause-and-effect, correlation doesn't imply causation, what if, right data and methods

5.2 Counterfactual evaluations

This section introduces counterfactual evaluation as estimating the outcomes of model decisions that were not made, using historical data while recognizing its limits.

Key terms: counterfactual evaluation, missing outcomes, logged policy, new policy, data quality, historical data, live experiments, A/B test

5.2.1 Anatomy of a counterfactual evaluation

The section identifies the data, estimation approach, metrics, and assumptions needed to evaluate a candidate policy using logged decisions.

Key terms: decision context, action, probability, observed outcome, candidate policy, baseline policy, inverse propensity weighting, expected value, metrics, accurate logs

5.2.2 The assumptions hiding underneath counterfactual evaluations

This section explains the assumptions that make counterfactual evaluations credible and why their results remain estimates rather than proof of real user impact.

Key terms: overlap, logged policy, new policy, propensity, context, decision time, logs, counterfactual evaluation, estimate, A/B test

5.2.3 Why counterfactual evaluations matter

Explains why counterfactual evaluations are useful when online A/B tests are impractical or risky.

Key terms: counterfactual evaluations, A/B test, historical data, detailed logs, retrospectively

5.3 What is counterfactual logging?

Counterfactual logging preserves production decision details so teams can estimate how a different policy might have performed, provided the logs contain useful evidence for that comparison.

Key terms: counterfactual logging, decision context, candidate set, action or slate selected, probability of that selection, policy version, overlap, effective sample size, propensities, logged policy

5.3.1 How counterfactual logging relates to causal inference

This section explains why counterfactual logging supplies the data needed to estimate the effects of decisions a model did not make.

Key terms: causal inference, counterfactual logging, alternative actions, probabilities, counterfactual outcomes

5.3.2 How much data is “enough”?

This section explains why counterfactual evaluation has no fixed data requirement and how to judge whether logged data supports a reliable estimate for a new policy.

Key terms: logged policy, new policy, policy overlap, action space size, exploration, outcome variance, confidence intervals, effective sample size, importance weights, clip large weights

5.3.3 Data alone is not enough

This section explains why counterfactual evaluation needs modeling in addition to logs, introduces three modeling approaches, and names the assumptions their estimates depend on.

Key terms: counterfactual evaluation, counterfactual modeling, unobserved outcomes, propensity score matching, inverse propensity weighting, causal machine learning, serving policy, accurate logging, sufficient overlap, logged and candidate policies

5.4 Estimating outcomes

The section contrasts counterfactual evaluation with label-based offline evaluation and distinguishes three related ways of thinking about unobserved outcomes.

Key terms: counterfactual evaluations, standard offline evaluations, decision policy, policy value, off-policy evaluation, incremental action value

5.4.1 Policy value

Defines policy value and explains how historical logs can estimate a candidate policy’s performance, including the weighting method and its limits.

Key terms: policy, policy value, context, action, candidate policy, logged policy, inverse propensity weighting, logged context, observed outcome, overlap

5.4.2 Off-policy evaluations

This section explains how off-policy evaluation uses historical logs and inverse propensity weighting to estimate a new policy’s performance without deploying it.

Key terms: off-policy evaluations (OPE), counterfactual logs, action probabilities, observed outcomes, contextual metadata, inverse propensity weighting (IPW), logged policy, new policy, overlap, importance weights

5.4.3 Designing incremental action value for your domain

This section explains how to design and check an incremental action value score around differences that matter between the actions available in a particular domain.

Key terms: incremental action value, domain hypotheses, historical data, scoring, discriminative power, action set, high vs. low incremental value predictions, actual user engagement

5.5 Logging best practices

This section introduces intentional logging: record the data a planned counterfactual evaluation needs rather than trying to log everything.

Key terms: log intentionally, counterfactual question, candidate set, propensities, context, outcomes

5.5.1 Log all possible actions

Explains what to log about the actions available at decision time so a counterfactual evaluation can consider reasonable alternatives.

Key terms: counterfactual evaluations, action spaces, candidate set, decision time, scores or probabilities, selected action

5.5.2 Record action probabilities

This subsection explains why counterfactual evaluations need logged probabilities for every possible action.

Key terms: action probabilities, probability distribution, inverse propensity weighting

5.5.3 Include contextual metadata

This subsection explains why counterfactual logs should include context so evaluations can account for differences in when and to whom an action was shown.

Key terms: counterfactual logging, contextual metadata, user characteristics, environmental factors, session details

5.5.4 Consistent and unbiased logs are the best types logs

This section explains why consistent, reliable, and unbiased logs are necessary for trustworthy counterfactual evaluations.

Key terms: consistent, unbiased logs, counterfactual evaluations, context

5.5.5 Practical example: search engine

This section uses a search-ranking example to show what logs must preserve for a reliable counterfactual evaluation.

Key terms: candidate set, selected ranking or slate, propensities, policy version, downstream outcomes, counterfactual evaluation

5.6 Strengths and realities of counterfactual evaluations

This section explains what counterfactual evaluations offer compared with A/B tests and why their estimates depend on reliable logs and modeling choices.

Key terms: counterfactual evaluations, A/B testing, logged data, historical data, action probabilities, logging pipelines, Inverse Propensity Weighting, IPW, Self-Normalized IPW, Clipped IPW

5.7 Common pitfalls in counterfactual logging

The section explains why reliable counterfactual evaluations depend on getting logging right, before introducing specific pitfalls.

Key terms: counterfactual logging, reliable evaluations

5.7.1 Incomplete data logging

This section explains how incomplete logging can make counterfactual and off-policy evaluations unreliable.

Key terms: counterfactual evaluations, off-policy evaluations, alternative actions, action probabilities, contextual data, inverse propensity weighting (IPW)

5.7.2 Biased propensity scores

This section explains how inaccurate propensity scores distort counterfactual evaluations and incremental action values, and how to check for that bias.

Key terms: propensity scores, incremental action value, incorrect weighting, possible actions, decision-making logic

5.7.3 Poor overlap between the logged policy and new policy

This section explains why poor overlap between logged and new policies makes counterfactual estimates unreliable, how to spot it, and ways to collect better evidence.

Key terms: poor overlap, logged policy, new policy, counterfactual evaluation, importance weights, effective sample size, unstable estimates, exploration, policy shift

5.7.4 Ignoring contextual variables

This section explains how missing contextual variables can distort policy value evaluations and why logs should capture relevant context while protecting user privacy.

Key terms: contextual variables, policy value, query metadata, context, metadata, sensitive user data, anonymize

5.8 Policy value evaluation for movie recommendations

Introduces a movie recommendation scenario for a forthcoming example of counterfactual logging and policy value evaluation.

5.8.1 The setup: data, design, and metrics

This section shows how logged recommendation data can be used to estimate the value of a new movie-recommendation policy.

Key terms: logged datasets, current recommendation policy, context data, outcomes, new policy, Inverse Propensity Weighting (IPW), original policy, policy value

5.8.2 Challenges along the way

This subsection identifies missing recommendation probabilities and large-scale logging as challenges in a movie recommendation evaluation.

Key terms: alternative recommendations, IPW adjustments, log events, data infrastructure and pipelines

5.8.3 Outcomes and insights

This subsection interprets a movie-recommendation off-policy evaluation through its overall results, weekend-user differences, and satisfaction signal.

Key terms: off-policy evaluation, new policy, contextual data, average viewing time, average user ratings

5.8.4 Making it actionable

The section draws an actionable lesson from the case: counterfactual logging and policy value evaluation can help assess new strategies without disrupting users, provided the work is carefully engineered and analyzed.

Key terms: counterfactual logging, policy value evaluation, possible actions, detailed context, actionable insights

5.9 LLM and agentic systems are tricker

Introduces counterfactual evaluation in LLM and agentic systems and previews why its setup needs more deliberate design than for traditional recommenders.

5.9.1 Why the action space gets messy fast

This section explains why counterfactual evaluation becomes harder when an agent makes a chain of decisions rather than one clear choice.

Key terms: action, decision point, counterfactual question, agentic system

5.9.2 Start with discrete, loggable decision points

The section shows how to make counterfactual evaluation of agents tractable by focusing on discrete decisions and logging each decision’s context, options, choice, and outcome.

Key terms: discrete, bounded, measurable outcomes, decision points, counterfactual evaluation, context at decision time, candidate options, selected action, probability or score, policy version

5.9.3 A practical example: an agent with tool routing

This section uses an agent’s tool-routing decision to show how to structure a counterfactual evaluation around a discrete, measurable choice.

Key terms: routing policy, candidate tools, IPW, downstream outcome, decision points, logging schema, dedicated router, propensities, outcome metrics

5.10 Engineering considerations

The section explains why the potential system load of counterfactual logging makes early engineering decisions important.

Key terms: counterfactual logging, counterfactual evaluations

5.10.1 Balancing system performance and logging

This section explains how to capture counterfactual logs without slowing the system that serves users.

Key terms: counterfactual logging, runtime operations, latency, asynchronously, dedicated logging service

5.10.2 Massive amounts of data

The section explains why counterfactual logging can require large-scale storage and why both data volume and quality matter for valid evaluations.

Key terms: counterfactual logging, alternative actions, contextual metadata, observed outcomes, distributed databases, cloud storage, sample size, statistically powered, data volume, data quality

5.11 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 4: Engineering system performance evaluations Chapter 6: Evaluating models in an A/B test →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.