Inverse propensity weighting (IPW) estimates how a candidate policy might perform using outcomes collected by an older policy. It corrects for the fact that the old policy chose some actions more often than others. The result is a counterfactual estimate, not a replay of what would actually have happened. section 5.11 section 5.4.2
To estimate a candidate policy’s value:
- Collect decision-time context, the candidate actions, the selected action, its probability under the old policy, and the observed outcome. section 5.11 section 5.4.1
- Replay each context through the candidate policy and calculate its probability of taking the action actually logged. section 5.4.1
- Weight the observed outcome by the candidate policy’s probability divided by the old policy’s probability for that same action. Aggregate the weighted outcomes to estimate policy value, then compare it with the baseline. section 5.4.1 section 5.2.1
Reweighting a recommendation
Suppose the old policy showed an item with probability 10%, and the candidate policy would show that same item in the same context with probability 30%. The logged outcome receives a weight of 3. If the candidate policy would never choose the logged action, that event contributes nothing to its estimate. The method reuses observed outcomes; it does not observe what would have happened for an unselected item. section 5.4.2 section 5.4.1
Reliable estimates require accurate logged probabilities, enough overlap between policies, and context that captures the important factors for comparison. A candidate action that rarely or never appears in the logs has little or no supporting evidence. Very small logged probabilities can also produce extreme weights and high variance. section 5.2.1 section 5.4.1 section 5.6
Check the estimate before using it for a product decision:
- Inspect weight distributions, effective sample size, confidence intervals, and results by segment. section 5.11
- Consider self-normalized IPW, which normalizes weights across the dataset, or clipped IPW, which caps extreme weights to improve stability. section 5.6
- Use offline estimates to explore candidates without exposing users to a change; use an A/B test to check whether the improvement occurs with real users. section 5.6
Listing 5.2 Calculating Off Policy Evaluation with multiple metrics
def evaluate_policy(self, metric='engagement_score'):
weights = self.calculate_importance_weights()
if metric == 'click_rate':
rewards = self.logged_data['clicked'].astype(float)
elif metric == 'watch_time':
rewards = self.logged_data['watch_time']
else: # engagement_score
rewards = (self.logged_data['clicked'].astype(float) + self.logged_data['watch_time'] / 100)
new_policy_value = np.sum(weights * rewards) / np.sum(weights) if np.sum(weights) > 0 else 0
original_policy_value = np.mean(rewards)
return {
'original_policy_value': original_policy_value,
'new_policy_value': new_policy_value,
'improvement': new_policy_value - original_policy_value,
'relative_improvement': ((new_policy_value / original_policy_value - 1) * 100 if original_policy_value > 0 else 0),
'effective_sample_size': (np.sum(weights) ** 2) / np.sum(weights ** 2) if np.sum(weights) > 0 else 0
}
This implementation compares the logged policy’s mean outcome with a self-normalized weighted estimate for click rate, watch time, or an engagement score. section 5.4.2
The same workflow can evaluate an agent’s tool-routing policy. Log the query, session context, candidate tools and their probabilities, the selected tool, and outcomes such as resolution or escalation. Then score the logged choices under the new router and apply IPW. section 5.9.3

