Chapter 5: Counterfactual evaluations
Introduces counterfactual evaluations as estimates of how an AI system might have performed had it made different decisions.
After this chapter you will be able to
- Counterfactual outcome estimates. Explain that counterfactual evaluations estimate how a different model or decision policy might have performed, rather than revealing what would certainly have happened.
- Cause versus coincidence. Explain why causal inference asks whether a decision caused an outcome rather than merely coinciding with it, using a product or recommendation example from the section.
- Estimating unchosen outcomes. Describe how causal inference can estimate what might have happened if a different decision had been made, and why that is useful for testing ideas without directly exposing users to them.
- Unobserved alternative outcomes. Explain that counterfactual evaluation asks what would have happened if a model had taken a different action, even though only the outcome of the action it actually took is observed.
- Limits of historical evidence. Describe why asking a counterfactual question does not guarantee a reliable answer: the estimate depends on how data was collected, whether the original policy explored alternatives, and whether the proposed policy is comparable to it.
- Decision Context in Logs. Describe what interaction data a counterfactual evaluation needs and why recording only the final outcome is insufficient.
- Candidate Policy Performance Estimate. Trace how logged observations can be used to estimate and compare a candidate policy’s performance.
- Conditions for Reliable Estimates. State the conditions the section says counterfactual estimates depend on and the consequence if they are not met.
- Overlap between policy actions. Explain why the logged policy must have had some chance of taking the actions the new policy would take, and what happens when it did not.
- Accurate logged propensities. Describe what a propensity records and why missing or inaccurate propensities undermine a counterfactual estimate.
- Decision-time context in logs. Explain why logs need the contextual variables that affected behavior at decision time.
- Estimates versus user impact. Distinguish what a counterfactual estimate can help decide from what requires an online A/B test.
- Value of counterfactual evaluations. Explain when counterfactual evaluations are useful and how historical data lets teams explore alternatives without exposing users to a new experience.
- Counterfactual decision logs. Explain what counterfactual logging records and how those records support a later comparison with a different policy.
- Useful evidence beyond data volume. Explain why a large counterfactual log may still provide little evidence for a particular new policy, using overlap and effective sample size.
- Logging for causal comparisons. Explain how counterfactual logging supports causal inference by recording alternative actions and their probabilities, so observed outcomes can be compared with estimates of what might have happened under a different decision.
- Policy-dependent data sufficiency. Explain why the amount of logged data needed depends on the new policy and the evaluation decision, rather than on a universal sample-size threshold. Describe how policy overlap, action space size, exploration, outcome variance, and desired precision affect that need.
- Estimate stability checks. Describe how teams assess whether an estimate from their existing logged data is stable enough to inform a decision.
- Missing counterfactual outcomes. Explain why detailed logs alone cannot establish what would have happened under a different decision or policy. Use the recommendation example to distinguish the observed click outcome from the unobserved outcome for the same user at the same moment.
- Unobserved policy outcomes. Explain that counterfactual evaluation estimates how a different decision policy might have performed, rather than checking predictions against known historical labels.
- Three outcome estimation ideas. Distinguish policy value, off-policy evaluation, and incremental action value by the roles assigned to each in this section.
- Policy and policy value. Explain what a policy is and what policy value estimates in a counterfactual evaluation.
- Logged outcome weighting. Trace how logged decisions are replayed through a candidate policy and weighted to estimate its value.
- Estimate reliability and overlap. Explain when a logged-data policy value estimate is credible and why inverse propensity weights can become unstable.
- Off-policy evaluation purpose and data. Explain what off-policy evaluation estimates, why it can be useful when an online experiment is impractical, and what historical data it requires.
- Inverse propensity weighting correction. Trace how inverse propensity weighting adjusts logged outcomes to better reflect the new policy, including the probability ratio, a simple numerical example, and its reliability conditions.
- Weighted policy value comparison. Explain how the example estimates and compares policy performance across reward metrics using weighted and unweighted outcomes.
- Factors that distinguish actions. Explain why incremental action value focuses on factors that distinguish available actions, rather than factors that merely predict outcomes overall. Describe how to use domain hypotheses and historical data to identify those factors.
- Scoring meaningful action contrasts. Describe how to weight factors in an incremental action value score and why their usefulness depends on the alternatives being compared.
- Intentional counterfactual logging. Explain why counterfactual logs should be designed around the evaluation question rather than indiscriminately collecting everything. Describe the kinds of information to consider recording, without needing to explain how estimators use them.
- Decision-Time Action Logging. Explain why counterfactual evaluations need records of reasonable alternatives to the selected action, and describe what to log when recording every possible action is impractical.
- Logged action probabilities. Explain why a system that makes random or weighted decisions must log the probability distribution over all possible actions for counterfactual evaluation.
- Contextual metadata in evaluation. Explain why counterfactual logging should capture contextual metadata, using the recommendation example and naming the kinds of context the section identifies.
- Consistent and unbiased logging. Explain how inconsistent context logging or over- and under-representation of actions or user groups can skew a counterfactual evaluation.
- Search ranking decision logs. Describe what a search engine should log when evaluating a new ranking algorithm offline, and explain why recording the full decision point matters for counterfactual questions.
- Counterfactual and A/B roles. Explain how counterfactual evaluations use historical logs to estimate outcomes for untested decisions without exposing users, and how A/B tests complement them by checking outcomes with real users.
- Logging and engineering limits. Describe how missing or inaccurate logged actions and action probabilities can make counterfactual estimates incomplete or biased, despite the speed and lower cost of reusing data.
- Logging reliability and evaluation accuracy. Explain why mistakes in counterfactual logging matter for the reliability of later evaluations.
- Incomplete logging and evaluation reliability. Explain why counterfactual evaluations need records of alternative actions, their probabilities, and relevant context—not just the action taken—and how missing information can distort off-policy evaluation (OPE).
- Propensity Score Bias Effects. Explain how inaccurate probabilities for possible actions can skew counterfactual comparisons, distort estimates of incremental action value, and lead to poor policy choices.
- Poor policy overlap. Explain how poor overlap arises when a new policy favors actions the logged policy rarely or never took, and why an estimate can be unreliable even when the evaluation produces a number.
- Overlap warning signs and responses. Describe the warning signs of poor overlap and explain why collecting more of the same data may not fix it.
- Contextual gaps in policy value. Explain how omitting context from logs can make a policy value evaluation misleading, including by mixing outcomes from different scenarios or hiding where a policy succeeds and fails.
- Logged recommendation data. Describe the logged actions, probabilities, context, and outcomes needed to evaluate a new movie-recommendation policy.
- New policy value estimation. Trace how the new policy is applied to logged sessions, how inverse propensity weighting adjusts outcomes, and how the weighted outcomes produce an overall policy value.
- Recommendation logging challenges. Explain how missing probabilities for alternative recommendations and the scale of logging can hinder the evaluation.
- Recommendation policy outcome interpretation. Explain the new policy’s reported gains in viewing time and user ratings, and describe why the weekend-user result makes contextual data important.
- Actionable counterfactual evaluation. Explain how the case uses counterfactual logging and policy value evaluation to assess new strategies without disrupting the user experience, and why the resulting insights still require careful engineering and analysis.
- Agentic counterfactual decision points. Explain why asking what an agent could have done differently requires identifying a particular decision point, and why evaluating the whole chain at once is usually a mistake.
- Tractable agent decision points. Explain why discrete, bounded agent decisions with measurable outcomes are practical targets for counterfactual evaluation, and give examples from the section.
- Decision point logging records. Describe what to record for each selected agent decision point so its choice and downstream outcome can be examined.
- Tool routing counterfactual evaluation. Trace how the example evaluates a new routing policy using logged decisions, the new policy’s probabilities, IPW, and downstream outcomes.
- Measurable agent decision points. Explain why the section focuses counterfactual evaluation on isolated, loggable routing decisions rather than every possible sentence, and describe the resulting design priorities.
- Counterfactual logging system load. Explain why early engineering decisions matter when implementing counterfactual evaluations: logging can place a heavy load on systems.
- Responsive counterfactual logging. Explain why logging everything during runtime can hurt the user experience and how decoupling logging from runtime operations addresses that problem.
- Counterfactual logging data volume. Explain why counterfactual logs grow quickly and what storage infrastructure may be needed.
- Data sufficiency and quality. Explain why a counterfactual evaluation may require a large sample and why volume alone does not establish validity.
What it builds on
- Why latency and load time really matter (chapter 4) · Why latency matters to the user experience
Section by section
Introduction
Introduces counterfactual evaluations as estimates of how an AI system might have performed had it made different decisions.
- Counterfactual outcome estimates: Explain that counterfactual evaluations estimate how a different model or decision policy might have performed, rather than revealing what would certainly have happened.
Why it matters: It sets the right expectation: an alternative decision can be evaluated, but its outcome cannot be observed with certainty.
Key terms: counterfactual evaluations, logged production data, decision probabilities, observed outcomes, assumptions
5.1 What is causal inference
The section introduces causal inference as a way to distinguish cause from coincidence and estimate what might have happened under a different decision.
- Cause versus coincidence: Explain why causal inference asks whether a decision caused an outcome rather than merely coinciding with it, using a product or recommendation example from the section.
Why it matters: It helps you avoid crediting a model change for an outcome it may not have caused. - Estimating unchosen outcomes: Describe how causal inference can estimate what might have happened if a different decision had been made, and why that is useful for testing ideas without directly exposing users to them.
Why it matters: It explains the value of asking “what if” when a product decision cannot simply be undone.
Key terms: causal inference, cause-and-effect, correlation doesn't imply causation, what if, right data and methods
5.2 Counterfactual evaluations
This section introduces counterfactual evaluation as estimating the outcomes of model decisions that were not made, using historical data while recognizing its limits.
- Unobserved alternative outcomes: Explain that counterfactual evaluation asks what would have happened if a model had taken a different action, even though only the outcome of the action it actually took is observed.
Why it matters: It distinguishes estimating an alternative outcome from measuring an outcome that actually occurred. - Limits of historical evidence: Describe why asking a counterfactual question does not guarantee a reliable answer: the estimate depends on how data was collected, whether the original policy explored alternatives, and whether the proposed policy is comparable to it.
Why it matters: It prevents an estimate from being treated as trustworthy merely because historical interaction data exists. - Offline evaluation advantage: Explain why estimating alternative outcomes from past decisions and their observed results can help evaluate a new model without first running a live experiment.
Why it matters: It explains the practical reason to attempt counterfactual evaluation before a live test.
Key terms: counterfactual evaluation, missing outcomes, logged policy, new policy, data quality, historical data, live experiments, A/B test
5.2.1 Anatomy of a counterfactual evaluation
The section identifies the data, estimation approach, metrics, and assumptions needed to evaluate a candidate policy using logged decisions.
- Decision Context in Logs: Describe what interaction data a counterfactual evaluation needs and why recording only the final outcome is insufficient.
Why it matters: A what-if estimate needs a record of how the original decision was made, not just how it ended. - Candidate Policy Performance Estimate: Trace how logged observations can be used to estimate and compare a candidate policy’s performance.
Why it matters: This is how the section connects recorded decisions to a comparison with a policy that was not used for those decisions. - Conditions for Reliable Estimates: State the conditions the section says counterfactual estimates depend on and the consequence if they are not met.
Why it matters: An estimated policy advantage is hard to trust when the logged data cannot support a fair comparison.
Key terms: decision context, action, probability, observed outcome, candidate policy, baseline policy, inverse propensity weighting, expected value, metrics, accurate logs
5.2.2 The assumptions hiding underneath counterfactual evaluations
This section explains the assumptions that make counterfactual evaluations credible and why their results remain estimates rather than proof of real user impact.
- Overlap between policy actions: Explain why the logged policy must have had some chance of taking the actions the new policy would take, and what happens when it did not.
Why it matters: Overlap determines whether past decisions offer evidence for the new policy’s choices. - Accurate logged propensities: Describe what a propensity records and why missing or inaccurate propensities undermine a counterfactual estimate.
Why it matters: An estimate can be misleading if the recorded action probabilities are wrong. - Decision-time context in logs: Explain why logs need the contextual variables that affected behavior at decision time.
Why it matters: Relevant context helps prevent comparisons between unlike decisions. - Estimates versus user impact: Distinguish what a counterfactual estimate can help decide from what requires an online A/B test.
Why it matters: A promising offline estimate should inform, not replace, a real-user test.
Key terms: overlap, logged policy, new policy, propensity, context, decision time, logs, counterfactual evaluation, estimate, A/B test
5.2.3 Why counterfactual evaluations matter
Explains why counterfactual evaluations are useful when online A/B tests are impractical or risky.
- Value of counterfactual evaluations: Explain when counterfactual evaluations are useful and how historical data lets teams explore alternatives without exposing users to a new experience.
Why it matters: This explains why teams might evaluate a change without first testing it on real users.
Key terms: counterfactual evaluations, A/B test, historical data, detailed logs, retrospectively
5.3 What is counterfactual logging?
Counterfactual logging preserves production decision details so teams can estimate how a different policy might have performed, provided the logs contain useful evidence for that comparison.
- Counterfactual decision logs: Explain what counterfactual logging records and how those records support a later comparison with a different policy.
Why it matters: An outcome alone does not preserve enough of the decision to ask what might have happened under another policy. - Useful evidence beyond data volume: Explain why a large counterfactual log may still provide little evidence for a particular new policy, using overlap and effective sample size.
Why it matters: The number of logged decisions can overstate how much evidence exists for the policy change you actually want to assess.
Key terms: counterfactual logging, decision context, candidate set, action or slate selected, probability of that selection, policy version, overlap, effective sample size, propensities, logged policy
5.3.1 How counterfactual logging relates to causal inference
This section explains why counterfactual logging supplies the data needed to estimate the effects of decisions a model did not make.
- Logging for causal comparisons: Explain how counterfactual logging supports causal inference by recording alternative actions and their probabilities, so observed outcomes can be compared with estimates of what might have happened under a different decision.
Why it matters: Logging alternatives makes it possible to ask how outcomes might have differed under another model decision.
Key terms: causal inference, counterfactual logging, alternative actions, probabilities, counterfactual outcomes
5.3.2 How much data is “enough”?
This section explains why counterfactual evaluation has no fixed data requirement and how to judge whether logged data supports a reliable estimate for a new policy.
- Policy-dependent data sufficiency: Explain why the amount of logged data needed depends on the new policy and the evaluation decision, rather than on a universal sample-size threshold. Describe how policy overlap, action space size, exploration, outcome variance, and desired precision affect that need.
Why it matters: A large log is not necessarily useful for evaluating a policy whose decisions the log rarely represents. - Estimate stability checks: Describe how teams assess whether an estimate from their existing logged data is stable enough to inform a decision.
Why it matters: Stability checks help keep a fragile estimate from driving a major production decision.
Key terms: logged policy, new policy, policy overlap, action space size, exploration, outcome variance, confidence intervals, effective sample size, importance weights, clip large weights
5.3.3 Data alone is not enough
This section explains why counterfactual evaluation needs modeling in addition to logs, introduces three modeling approaches, and names the assumptions their estimates depend on.
- Missing counterfactual outcomes: Explain why detailed logs alone cannot establish what would have happened under a different decision or policy. Use the recommendation example to distinguish the observed click outcome from the unobserved outcome for the same user at the same moment.
Why it matters: It clarifies the question a counterfactual evaluation must answer beyond reporting logged results. - Counterfactual modeling approaches: Describe how propensity score matching, inverse propensity weighting, and causal machine learning approach the estimation of outcomes under another action or policy, at the level introduced here.
Why it matters: These approaches show what it means to turn logged observations into estimates about another policy. - Conditions for credible estimates: State the assumptions the section says counterfactual methods depend on, and explain why recording data does not remove the need for them.
Why it matters: It prevents treating a modeled estimate as reliable merely because data was logged and a technique was applied.
Key terms: counterfactual evaluation, counterfactual modeling, unobserved outcomes, propensity score matching, inverse propensity weighting, causal machine learning, serving policy, accurate logging, sufficient overlap, logged and candidate policies
5.4 Estimating outcomes
The section contrasts counterfactual evaluation with label-based offline evaluation and distinguishes three related ways of thinking about unobserved outcomes.
- Unobserved policy outcomes: Explain that counterfactual evaluation estimates how a different decision policy might have performed, rather than checking predictions against known historical labels.
Why it matters: It clarifies why historical results cannot directly answer the question counterfactual evaluation asks. - Three outcome estimation ideas: Distinguish policy value, off-policy evaluation, and incremental action value by the roles assigned to each in this section.
Why it matters: Keeping the three roles separate makes the outcome-estimation question easier to follow.
Key terms: counterfactual evaluations, standard offline evaluations, decision policy, policy value, off-policy evaluation, incremental action value
5.4.1 Policy value
Defines policy value and explains how historical logs can estimate a candidate policy’s performance, including the weighting method and its limits.
- Policy and policy value: Explain what a policy is and what policy value estimates in a counterfactual evaluation.
Why it matters: This identifies what the evaluation is trying to estimate. - Logged outcome weighting: Trace how logged decisions are replayed through a candidate policy and weighted to estimate its value.
Why it matters: Weighting lets historical outcomes provide evidence about a policy that has not yet been deployed. - Estimate reliability and overlap: Explain when a logged-data policy value estimate is credible and why inverse propensity weights can become unstable.
Why it matters: A calculated policy value is not automatically trustworthy simply because historical data exists.
Key terms: policy, policy value, context, action, candidate policy, logged policy, inverse propensity weighting, logged context, observed outcome, overlap
5.4.2 Off-policy evaluations
This section explains how off-policy evaluation uses historical logs and inverse propensity weighting to estimate a new policy’s performance without deploying it.
- Off-policy evaluation purpose and data: Explain what off-policy evaluation estimates, why it can be useful when an online experiment is impractical, and what historical data it requires.
Why it matters: The estimate is only as useful as the logged decisions and outcomes it can draw on. - Inverse propensity weighting correction: Trace how inverse propensity weighting adjusts logged outcomes to better reflect the new policy, including the probability ratio, a simple numerical example, and its reliability conditions.
Why it matters: The weighting corrects for the old policy’s preferences rather than treating its logged actions as equally representative of the new one. - Movie policy action probabilities: Describe how the example’s quality-focused movie policy assigns recommendation probabilities and why its initial scores are not yet probabilities.
Why it matters: The example shows where the new-policy probabilities used in OPE come from. - Weighted policy value comparison: Explain how the example estimates and compares policy performance across reward metrics using weighted and unweighted outcomes.
Why it matters: This calculation connects the probability correction to a measurable policy comparison. - Counterfactual estimate reliability checks: Explain why a favorable OPE point estimate may still be untrustworthy and what the section says to inspect before relying on it.
Why it matters: A promising estimated lift is less persuasive if it rests on very little effectively useful data.
Key terms: off-policy evaluations (OPE), counterfactual logs, action probabilities, observed outcomes, contextual metadata, inverse propensity weighting (IPW), logged policy, new policy, overlap, importance weights
5.4.3 Designing incremental action value for your domain
This section explains how to design and check an incremental action value score around differences that matter between the actions available in a particular domain.
- Factors that distinguish actions: Explain why incremental action value focuses on factors that distinguish available actions, rather than factors that merely predict outcomes overall. Describe how to use domain hypotheses and historical data to identify those factors.
Why it matters: A strong predictor of engagement may add little to a decision if it does not distinguish the available options. - Scoring meaningful action contrasts: Describe how to weight factors in an incremental action value score and why their usefulness depends on the alternatives being compared.
Why it matters: The score should reflect differences a decision can actually make, not just attributes that sound important in isolation. - Historical checks of predicted differences: Describe how to test whether predicted incremental value differences correspond to observed differences in historical outcomes.
Why it matters: Checking predicted contrasts against observed engagement helps reveal whether the scoring logic captures meaningful differences.
Key terms: incremental action value, domain hypotheses, historical data, scoring, discriminative power, action set, high vs. low incremental value predictions, actual user engagement
5.5 Logging best practices
This section introduces intentional logging: record the data a planned counterfactual evaluation needs rather than trying to log everything.
- Intentional counterfactual logging: Explain why counterfactual logs should be designed around the evaluation question rather than indiscriminately collecting everything. Describe the kinds of information to consider recording, without needing to explain how estimators use them.
Why it matters: Planning logs around an evaluation helps retain useful evidence without collecting data indiscriminately.
Key terms: log intentionally, counterfactual question, candidate set, propensities, context, outcomes
5.5.1 Log all possible actions
Explains what to log about the actions available at decision time so a counterfactual evaluation can consider reasonable alternatives.
- Decision-Time Action Logging: Explain why counterfactual evaluations need records of reasonable alternatives to the selected action, and describe what to log when recording every possible action is impractical.
Why it matters: Without a record of reasonable alternatives, it is harder to ask what might have happened had the system chosen differently.
Key terms: counterfactual evaluations, action spaces, candidate set, decision time, scores or probabilities, selected action
5.5.2 Record action probabilities
This subsection explains why counterfactual evaluations need logged probabilities for every possible action.
- Logged action probabilities: Explain why a system that makes random or weighted decisions must log the probability distribution over all possible actions for counterfactual evaluation.
Why it matters: Recording what could have been chosen, and how likely each choice was, makes later outcome comparisons more reliable.
Key terms: action probabilities, probability distribution, inverse propensity weighting
5.5.3 Include contextual metadata
This subsection explains why counterfactual logs should include context so evaluations can account for differences in when and to whom an action was shown.
- Contextual metadata in evaluation: Explain why counterfactual logging should capture contextual metadata, using the recommendation example and naming the kinds of context the section identifies.
Why it matters: Without the circumstances around a logged recommendation, differences in its observed performance are harder to account for.
Key terms: counterfactual logging, contextual metadata, user characteristics, environmental factors, session details
5.5.4 Consistent and unbiased logs are the best types logs
This section explains why consistent, reliable, and unbiased logs are necessary for trustworthy counterfactual evaluations.
- Consistent and unbiased logging: Explain how inconsistent context logging or over- and under-representation of actions or user groups can skew a counterfactual evaluation.
Why it matters: An evaluation can appear credible while reflecting gaps or imbalances in its logs rather than the outcomes you meant to assess.
Key terms: consistent, unbiased logs, counterfactual evaluations, context
5.5.5 Practical example: search engine
This section uses a search-ranking example to show what logs must preserve for a reliable counterfactual evaluation.
- Search ranking decision logs: Describe what a search engine should log when evaluating a new ranking algorithm offline, and explain why recording the full decision point matters for counterfactual questions.
Why it matters: These records let you assess an alternative ranking against the decision users actually encountered.
Key terms: candidate set, selected ranking or slate, propensities, policy version, downstream outcomes, counterfactual evaluation
5.6 Strengths and realities of counterfactual evaluations
This section explains what counterfactual evaluations offer compared with A/B tests and why their estimates depend on reliable logs and modeling choices.
- Counterfactual and A/B roles: Explain how counterfactual evaluations use historical logs to estimate outcomes for untested decisions without exposing users, and how A/B tests complement them by checking outcomes with real users.
Why it matters: This distinction helps you use offline estimates to explore ideas without mistaking them for observed user outcomes. - Logging and engineering limits: Describe how missing or inaccurate logged actions and action probabilities can make counterfactual estimates incomplete or biased, despite the speed and lower cost of reusing data.
Why it matters: Fast, inexpensive estimates are useful only when the records behind them can support the comparison. - Estimation method trade-offs: Distinguish the modeling trade-offs described here: small propensity scores can make IPW estimates high-variance, stabilized IPW variants address extreme weights, and Direct Method estimates depend on accurate outcome predictions and data coverage.
Why it matters: A plausible what-if number is not automatically reliable; the estimation method has to fit the available data.
Key terms: counterfactual evaluations, A/B testing, logged data, historical data, action probabilities, logging pipelines, Inverse Propensity Weighting, IPW, Self-Normalized IPW, Clipped IPW
5.7 Common pitfalls in counterfactual logging
The section explains why reliable counterfactual evaluations depend on getting logging right, before introducing specific pitfalls.
- Logging reliability and evaluation accuracy: Explain why mistakes in counterfactual logging matter for the reliability of later evaluations.
Why it matters: The quality of an evaluation depends on the logging it relies on.
Key terms: counterfactual logging, reliable evaluations
5.7.1 Incomplete data logging
This section explains how incomplete logging can make counterfactual and off-policy evaluations unreliable.
- Incomplete logging and evaluation reliability: Explain why counterfactual evaluations need records of alternative actions, their probabilities, and relevant context—not just the action taken—and how missing information can distort off-policy evaluation (OPE).
Why it matters: An unreliable OPE can mispredict a new policy’s performance and lead to costly deployment decisions.
Key terms: counterfactual evaluations, off-policy evaluations, alternative actions, action probabilities, contextual data, inverse propensity weighting (IPW)
5.7.2 Biased propensity scores
This section explains how inaccurate propensity scores distort counterfactual evaluations and incremental action values, and how to check for that bias.
- Propensity Score Bias Effects: Explain how inaccurate probabilities for possible actions can skew counterfactual comparisons, distort estimates of incremental action value, and lead to poor policy choices.
Why it matters: A policy change can look promising in the evaluation because of bad probability estimates rather than real improvement. - Propensity Score Validation: Describe the checks this section recommends to reduce bias in propensity scores.
Why it matters: These checks help catch probability estimates that would otherwise make the evaluation misleading.
Key terms: propensity scores, incremental action value, incorrect weighting, possible actions, decision-making logic
5.7.3 Poor overlap between the logged policy and new policy
This section explains why poor overlap between logged and new policies makes counterfactual estimates unreliable, how to spot it, and ways to collect better evidence.
- Poor policy overlap: Explain how poor overlap arises when a new policy favors actions the logged policy rarely or never took, and why an estimate can be unreliable even when the evaluation produces a number.
Why it matters: A numerical estimate is not necessarily well supported by the historical evidence. - Overlap warning signs and responses: Describe the warning signs of poor overlap and explain why collecting more of the same data may not fix it.
Why it matters: These checks help you avoid acting on an estimate that hinges on a handful of observations.
Key terms: poor overlap, logged policy, new policy, counterfactual evaluation, importance weights, effective sample size, unstable estimates, exploration, policy shift
5.7.4 Ignoring contextual variables
This section explains how missing contextual variables can distort policy value evaluations and why logs should capture relevant context while protecting user privacy.
- Contextual gaps in policy value: Explain how omitting context from logs can make a policy value evaluation misleading, including by mixing outcomes from different scenarios or hiding where a policy succeeds and fails.
Why it matters: An apparently strong overall result may conceal a policy that performs poorly in important situations. - Contextual logging and privacy: Describe what context to capture alongside model outcomes and the privacy precaution the section calls for.
Why it matters: Useful context makes evaluations more interpretable, but collecting it must not disregard user privacy.
Key terms: contextual variables, policy value, query metadata, context, metadata, sensitive user data, anonymize
5.8 Policy value evaluation for movie recommendations
Introduces a movie recommendation scenario for a forthcoming example of counterfactual logging and policy value evaluation.
5.8.1 The setup: data, design, and metrics
This section shows how logged recommendation data can be used to estimate the value of a new movie-recommendation policy.
- Logged recommendation data: Describe the logged actions, probabilities, context, and outcomes needed to evaluate a new movie-recommendation policy.
Why it matters: These logs provide both the decisions and the observed responses needed for the comparison. - New policy value estimation: Trace how the new policy is applied to logged sessions, how inverse propensity weighting adjusts outcomes, and how the weighted outcomes produce an overall policy value.
Why it matters: Weighting lets the evaluation estimate a new policy's value from data collected under the original policy.
Key terms: logged datasets, current recommendation policy, context data, outcomes, new policy, Inverse Propensity Weighting (IPW), original policy, policy value
5.8.2 Challenges along the way
This subsection identifies missing recommendation probabilities and large-scale logging as challenges in a movie recommendation evaluation.
- Recommendation logging challenges: Explain how missing probabilities for alternative recommendations and the scale of logging can hinder the evaluation.
Why it matters: These challenges affect whether the evaluation has the information and timely data it needs.
Key terms: alternative recommendations, IPW adjustments, log events, data infrastructure and pipelines
5.8.3 Outcomes and insights
This subsection interprets a movie-recommendation off-policy evaluation through its overall results, weekend-user differences, and satisfaction signal.
- Recommendation policy outcome interpretation: Explain the new policy’s reported gains in viewing time and user ratings, and describe why the weekend-user result makes contextual data important.
Why it matters: Context and a second outcome metric make the policy’s apparent improvement more informative than viewing time alone.
Key terms: off-policy evaluation, new policy, contextual data, average viewing time, average user ratings
5.8.4 Making it actionable
The section draws an actionable lesson from the case: counterfactual logging and policy value evaluation can help assess new strategies without disrupting users, provided the work is carefully engineered and analyzed.
- Actionable counterfactual evaluation: Explain how the case uses counterfactual logging and policy value evaluation to assess new strategies without disrupting the user experience, and why the resulting insights still require careful engineering and analysis.
Why it matters: It connects the evaluation setup to decisions about improving a system while keeping its engineering demands in view.
Key terms: counterfactual logging, policy value evaluation, possible actions, detailed context, actionable insights
5.9 LLM and agentic systems are tricker
Introduces counterfactual evaluation in LLM and agentic systems and previews why its setup needs more deliberate design than for traditional recommenders.
5.9.1 Why the action space gets messy fast
This section explains why counterfactual evaluation becomes harder when an agent makes a chain of decisions rather than one clear choice.
- Agentic counterfactual decision points: Explain why asking what an agent could have done differently requires identifying a particular decision point, and why evaluating the whole chain at once is usually a mistake.
Why it matters: Pinpointing the decision in question makes an agentic counterfactual more meaningful.
Key terms: action, decision point, counterfactual question, agentic system
5.9.2 Start with discrete, loggable decision points
The section shows how to make counterfactual evaluation of agents tractable by focusing on discrete decisions and logging each decision’s context, options, choice, and outcome.
- Tractable agent decision points: Explain why discrete, bounded agent decisions with measurable outcomes are practical targets for counterfactual evaluation, and give examples from the section.
Why it matters: Choosing a bounded decision gives you a practical unit of agent behavior to evaluate. - Decision point logging records: Describe what to record for each selected agent decision point so its choice and downstream outcome can be examined.
Why it matters: A record of the available choices and what followed makes the decision point usable for evaluation.
Key terms: discrete, bounded, measurable outcomes, decision points, counterfactual evaluation, context at decision time, candidate options, selected action, probability or score, policy version
5.9.3 A practical example: an agent with tool routing
This section uses an agent’s tool-routing decision to show how to structure a counterfactual evaluation around a discrete, measurable choice.
- Tool routing counterfactual evaluation: Trace how the example evaluates a new routing policy using logged decisions, the new policy’s probabilities, IPW, and downstream outcomes.
Why it matters: It shows what must be recorded to compare a proposed routing policy with the one that made the logged decisions. - Measurable agent decision points: Explain why the section focuses counterfactual evaluation on isolated, loggable routing decisions rather than every possible sentence, and describe the resulting design priorities.
Why it matters: Choosing a measurable decision point determines what the logging pipeline and outcome measures need to capture.
Key terms: routing policy, candidate tools, IPW, downstream outcome, decision points, logging schema, dedicated router, propensities, outcome metrics
5.10 Engineering considerations
The section explains why the potential system load of counterfactual logging makes early engineering decisions important.
- Counterfactual logging system load: Explain why early engineering decisions matter when implementing counterfactual evaluations: logging can place a heavy load on systems.
Why it matters: It connects the demands of logging to the need to plan the supporting system early.
Key terms: counterfactual logging, counterfactual evaluations
5.10.1 Balancing system performance and logging
This section explains how to capture counterfactual logs without slowing the system that serves users.
- Responsive counterfactual logging: Explain why logging everything during runtime can hurt the user experience and how decoupling logging from runtime operations addresses that problem.
Why it matters: Useful evaluation logs should not come at the cost of making users wait for results.
Key terms: counterfactual logging, runtime operations, latency, asynchronously, dedicated logging service
5.10.2 Massive amounts of data
The section explains why counterfactual logging can require large-scale storage and why both data volume and quality matter for valid evaluations.
- Counterfactual logging data volume: Explain why counterfactual logs grow quickly and what storage infrastructure may be needed.
Why it matters: Planning for the full contents of each log entry helps avoid underestimating storage needs. - Data sufficiency and quality: Explain why a counterfactual evaluation may require a large sample and why volume alone does not establish validity.
Why it matters: Having enough records is useful only if the evaluation can support a valid decision.
Key terms: counterfactual logging, alternative actions, contextual metadata, observed outcomes, distributed databases, cloud storage, sample size, statistically powered, data volume, data quality
5.11 Summary
Where this is used later
- Guarding against variance inflation (chapter 8) · The idea of a counterfactual evaluation
- A/B testing (chapter 6) · Motivates A/B testing when real user impact needs to be measured.
- Feedback loops unique to AI models (chapter 8) · the use of logged data in counterfactual evaluation
Concepts explained in this chapter
- policyA decision rule that maps a context to an action or a probability distribution over actions.
- inverse propensity weightingInverse propensity scoring estimates counterfactual outcomes by weighting observed outcomes inversely by the probabilities that generated their actions.
- counterfactual reasoningCounterfactual evaluation estimates outcomes under decisions or policies other than those that generated the observed data.
- policy overlapThe condition that a candidate policy's relevant actions have positive probability under the policy that generated the logged data.
- effective sample sizeEffective sample size is the amount of statistically usable evidence remaining after observations are weighted or otherwise adjusted.
- off-policy evaluationOff-policy evaluation estimates the performance of a target policy using data collected under a different behavior policy.
- probability theoryProbability is a numerical measure from zero to one representing the likelihood of an event or choice.
- policy valuePolicy value is the expected performance of a decision-making policy if deployed under the evaluation conditions.
- importance weightAn importance weight is a factor that reweights an observation to account for a difference between the distribution that generated it and a target distribution.
- logLogging is the practice of recording detailed information about system behavior, user interactions, and outcomes.
- counterfactual loggingCounterfactual logging records observed interactions together with information needed to estimate what would have occurred under alternative experiment variants or policies.
- expected rewardExpected reward is the predicted average benefit of a policy over its possible actions and outcomes.
- action spaceAn action space is the set of actions available to a policy or decision-making system in a given context.
- contextA contextual feature is an input variable describing the circumstances surrounding a user's current interaction or request.
- causal inferenceCausal analysis estimates the effects of actions or interventions on outcomes while accounting for how observations were generated.
- candidate setA candidate set is the collection of items or actions considered for selection by a decision-making system.
- incremental action valueThe estimated change in outcome attributable to taking a particular alternative action instead of the observed or reference action.
- language modelA neural language model trained on large text corpora to understand, generate, or transform natural-language content.
- historical dataHistorical interaction data records past user actions and engagement events in a product.
- propensity scoreA propensity score is an estimated probability that a logging policy selects a particular action for a given context.
- AI agentAn agentic system is an AI system that performs tasks through multi-step actions, often by deciding when and how to invoke tools.
- versionA model version is a distinct released or identified state of a model with a specific implementation, parameters, or training artifacts.
- correlationCorrelation is a statistical association between variables that does not by itself establish that one variable causes another.
- baselineA baseline is an existing model, method, heuristic, or reference result against which a new approach is compared.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
