An importance weight adjusts an observed outcome for a difference between the policy that generated it and a target policy. In off-policy evaluation, the weight is the probability that the new policy would take the observed action divided by the probability that the logged policy took it. Weighted outcomes help estimate how the new policy would perform using existing data. section 5.4.2
Reweighting a recommendation
If the logged policy showed an item 10% of the time and the new policy would show it 30% of the time, the observed outcome gets a weight of 3. The adjusted outcomes can then contribute to an estimate of expected reward or another policy metric. section 5.4.2
Listing 5.2 Calculating Off Policy Evaluation with multiple metrics
def evaluate_policy(self, metric='engagement_score'):
weights = self.calculate_importance_weights()
if metric == 'click_rate':
rewards = self.logged_data['clicked'].astype(float)
elif metric == 'watch_time':
rewards = self.logged_data['watch_time']
else: # engagement_score
rewards = (self.logged_data['clicked'].astype(float) + self.logged_data['watch_time'] / 100)
new_policy_value = np.sum(weights * rewards) / np.sum(weights) if np.sum(weights) > 0 else 0
original_policy_value = np.mean(rewards)
return {
'original_policy_value': original_policy_value,
'new_policy_value': new_policy_value,
'improvement': new_policy_value - original_policy_value,
'relative_improvement': ((new_policy_value / original_policy_value - 1) * 100 if original_policy_value > 0 else 0),
'effective_sample_size': (np.sum(weights) ** 2) / np.sum(weights ** 2) if np.sum(weights) > 0 else 0
}
The evaluation code applies importance weights to click rate, watch time, or an engagement score, then divides the weighted reward sum by the sum of weights. section 5.4.2
Reliable weighting depends on accurate logged probabilities and sufficient overlap between policies. Limited exploration, a large action space, or policies that differ substantially can leave too little useful evidence. Inspect the weight distribution, effective sample size, and confidence intervals; if clipping large weights or changing the time window makes the estimate swing wildly, it may not support a high-confidence decision. section 5.4.2 section 5.3.2




