liveBook · Manning

Counterfactual logging

As explained in AI Model Evaluation

Counterfactual logging records observed interactions together with information needed to estimate what would have occurred under alternative experiment variants or policies.

Counterfactual logging records a production decision and its outcome, along with enough information to estimate how a different policy might have performed. It supports offline evaluation before exposing users to a change and can help diagnose whether an online experiment produced a meaningful difference. The alternative outcome is an estimate under assumptions, not an observed replay of another reality. section 5.3 section 2.1.3 section 5.11

Capture the decision as it looked at the time:

  • The decision context and candidate set, such as the user and session state and the alternatives considered. section 5.3
  • The selected action or slate, its selection probability, and, when possible, the probability distribution over candidates. section 5.3
  • The policy or model version and the observed outcome, such as a click, watch, purchase, or task completion. section 5.3

A recommendation decision

Suppose a system recommends a movie and the user watches it. Log the candidates, selected recommendation, selection probability, relevant context, policy version, and watch outcome. Those records can help estimate how a different recommendation policy might perform; they do not reveal whether this particular user would have watched another movie. section 5.3 section 5.3.1 section 5.11

Useful evaluation depends on more than log volume. If the original policy rarely explores alternatives, propensities are missing, or the new policy chooses very different actions, even millions of records may provide little useful evidence. Check overlap, effective sample size, confidence intervals, segment-level results, and weight distributions. In experiments, use consistent randomization keys so assignment and logging behave consistently across the simulated comparison. section 5.3 section 5.11 section 2.1.3

Logging can add latency and generate substantial storage costs. Consider asynchronous logging or a dedicated logging service to keep the product responsive, and plan storage for action probabilities, context, and multiple outcome events. Balance evaluation needs with privacy, governance, and maintainability. section 5.10.1 section 5.10.2 section 5.11

Table 2.3 Common Simulation Evaluations
Type of Simulation Evaluation
Purpose
Example
Logged Replay Evaluations
Replay historical events to evaluate how a new model or system would behave based on past interactions.
Replay past search queries or recommendation requests and compare the new ranking output against historical engagement signals.
Counterfactual / Off-Policy Evaluations
Estimate how an alternative action, recommendation, ranking policy, or decision might have performed using logged data.
Estimate how promoting item A instead of item B might have affected clicks or purchases, assuming the right logging and propensity information are available.
Scenario-Based Stress Tests
Test model behavior under rare, extreme, or intentionally challenging conditions.
Evaluate a recommender on cold-start users, sparse histories, sudden catalog changes, missing features, or traffic spikes.
Synthetic Prompt Evaluations
Test LLM-powered features using scripted or synthetically generated prompts that represent edge cases or underrepresented scenarios.
Evaluate whether a recommendation explanation model stays grounded when given sparse watch history, ambiguous preferences, or adversarial prompts.
Agent Environment Simulations
Evaluate how an agent behaves when it must plan, call tools, handle errors, or complete a workflow in a controlled environment.
Test whether a support agent calls the right ticket lookup tool, handles a failed API response, and still drafts an accurate customer response.
Simulated User Evaluations
Simulate user interactions to test how a system behaves across multi-turn flows.
Evaluate whether an LLM-powered movie assistant can handle follow-up requests like “nothing scary,” “under 90 minutes,” and “something my kids can watch.”

Compare counterfactual evaluation with logged replay and other simulation approaches. section 2.1.3

How AI Model Evaluation covers it

How deep it goes
Specifies decision-time fields, reliability checks, and architecture trade-offs without prescribing a concrete implementation.
Examples
Uses recommendation decisions and observed engagement to illustrate estimating outcomes under an alternative policy.
What you can do after
Evaluate and understand alternative policies using logged decisions while checking overlap and estimate reliability.
How it connects
Connects causal inference and off-policy evaluation to propensities, policy value, and inverse propensity weighting.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.