Effective sample size describes how much usable evidence remains after data is weighted. In counterfactual evaluation, a large log can still provide little evidence for a particular policy comparison if the logged policy rarely explored alternatives or the new policy makes very different decisions. section 5.3 section 5.3.2
Off-policy evaluation weights logged outcomes according to how likely the new policy was to take the logged actions. If a few sessions carry most of the weight, the estimate may be fragile even when the original dataset is large. Effective sample size helps assess that risk, but it is not sufficient on its own. section 5.4.2 section 5.11
Before using a counterfactual estimate for a decision, check:
- Overlap and logging: Did the logged policy sometimes take the actions the new policy would take, and were action probabilities recorded? section 5.3
- Effective sample size and weights: Is usable evidence only a small fraction of the logged data, or do a few sessions carry most of the weight? section 5.3.2 section 5.4.2
- Uncertainty and stability: Inspect confidence intervals and see whether the estimate changes substantially when large weights are clipped, a segment is removed, or the time window changes. section 5.3.2
Comparing recommendation policies
A logged movie recommendation policy provides observed rewards for sessions. A new policy’s estimated value is the weighted average of those rewards; the original policy’s performance is their unweighted average. An estimated 8% improvement is less convincing if it depends on a tiny number of heavily weighted sessions. section 5.4.2
Listing 5.2 Calculating Off Policy Evaluation with multiple metrics
def evaluate_policy(self, metric='engagement_score'):
weights = self.calculate_importance_weights()
if metric == 'click_rate':
rewards = self.logged_data['clicked'].astype(float)
elif metric == 'watch_time':
rewards = self.logged_data['watch_time']
else: # engagement_score
rewards = (self.logged_data['clicked'].astype(float) + self.logged_data['watch_time'] / 100)
new_policy_value = np.sum(weights * rewards) / np.sum(weights) if np.sum(weights) > 0 else 0
original_policy_value = np.mean(rewards)
return {
'original_policy_value': original_policy_value,
'new_policy_value': new_policy_value,
'improvement': new_policy_value - original_policy_value,
'relative_improvement': ((new_policy_value / original_policy_value - 1) * 100 if original_policy_value > 0 else 0),
'effective_sample_size': (np.sum(weights) ** 2) / np.sum(weights ** 2) if np.sum(weights) > 0 else 0
}
Inspect how the evaluation selects rewards and compares weighted and unweighted policy values. section 5.4.2

