liveBook · Manning

Inverse propensity weighting

As explained in AI Model Evaluation

Inverse propensity scoring estimates counterfactual outcomes by weighting observed outcomes inversely by the probabilities that generated their actions.

Inverse propensity weighting (IPW) estimates how a candidate policy might perform using outcomes collected by an older policy. It corrects for the fact that the old policy chose some actions more often than others. The result is a counterfactual estimate, not a replay of what would actually have happened. section 5.11 section 5.4.2

To estimate a candidate policy’s value:

  • Collect decision-time context, the candidate actions, the selected action, its probability under the old policy, and the observed outcome. section 5.11 section 5.4.1
  • Replay each context through the candidate policy and calculate its probability of taking the action actually logged. section 5.4.1
  • Weight the observed outcome by the candidate policy’s probability divided by the old policy’s probability for that same action. Aggregate the weighted outcomes to estimate policy value, then compare it with the baseline. section 5.4.1 section 5.2.1

Reweighting a recommendation

Suppose the old policy showed an item with probability 10%, and the candidate policy would show that same item in the same context with probability 30%. The logged outcome receives a weight of 3. If the candidate policy would never choose the logged action, that event contributes nothing to its estimate. The method reuses observed outcomes; it does not observe what would have happened for an unselected item. section 5.4.2 section 5.4.1

Reliable estimates require accurate logged probabilities, enough overlap between policies, and context that captures the important factors for comparison. A candidate action that rarely or never appears in the logs has little or no supporting evidence. Very small logged probabilities can also produce extreme weights and high variance. section 5.2.1 section 5.4.1 section 5.6

Check the estimate before using it for a product decision:

  • Inspect weight distributions, effective sample size, confidence intervals, and results by segment. section 5.11
  • Consider self-normalized IPW, which normalizes weights across the dataset, or clipped IPW, which caps extreme weights to improve stability. section 5.6
  • Use offline estimates to explore candidates without exposing users to a change; use an A/B test to check whether the improvement occurs with real users. section 5.6
Listing 5.2 Calculating Off Policy Evaluation with multiple metrics
def evaluate_policy(self, metric='engagement_score'):
        weights = self.calculate_importance_weights()
        
        if metric == 'click_rate':
            rewards = self.logged_data['clicked'].astype(float)
        elif metric == 'watch_time':
            rewards = self.logged_data['watch_time']
        else:  # engagement_score
            rewards = (self.logged_data['clicked'].astype(float) + self.logged_data['watch_time'] / 100)
        
        new_policy_value = np.sum(weights * rewards) / np.sum(weights) if np.sum(weights) > 0 else 0
        original_policy_value = np.mean(rewards)
        
        return {
            'original_policy_value': original_policy_value,
            'new_policy_value': new_policy_value,
            'improvement': new_policy_value - original_policy_value,
            'relative_improvement': ((new_policy_value / original_policy_value - 1) * 100 if original_policy_value > 0 else 0),
            'effective_sample_size': (np.sum(weights) ** 2) / np.sum(weights ** 2) if np.sum(weights) > 0 else 0
        }
 

This implementation compares the logged policy’s mean outcome with a self-normalized weighted estimate for click rate, watch time, or an engagement score. section 5.4.2

The same workflow can evaluate an agent’s tool-routing policy. Log the query, session context, candidate tools and their probabilities, the selected tool, and outcomes such as resolution or escalation. Then score the logged choices under the new router and apply IPW. section 5.9.3

How AI Model Evaluation covers it

How deep it goes
Works through probability ratios, aggregation, assumptions, and stability checks, with a multi-metric implementation.
Examples
Uses item recommendation probabilities, movie engagement metrics, and an agent’s tool-routing decisions.
What you can do after
Evaluate and compare candidate policies offline, check estimate reliability, and identify results for online confirmation.
How it connects
Connects counterfactual evaluation, off-policy evaluation, policy value, and A/B testing.

Also explained in

Compare how 2 books explain inverse propensity weighting →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.