liveBook · Manning

Effective sample size

As explained in AI Model Evaluation

Effective sample size is the amount of statistically usable evidence remaining after observations are weighted or otherwise adjusted.

Effective sample size describes how much usable evidence remains after data is weighted. In counterfactual evaluation, a large log can still provide little evidence for a particular policy comparison if the logged policy rarely explored alternatives or the new policy makes very different decisions. section 5.3 section 5.3.2

Off-policy evaluation weights logged outcomes according to how likely the new policy was to take the logged actions. If a few sessions carry most of the weight, the estimate may be fragile even when the original dataset is large. Effective sample size helps assess that risk, but it is not sufficient on its own. section 5.4.2 section 5.11

Before using a counterfactual estimate for a decision, check:

  • Overlap and logging: Did the logged policy sometimes take the actions the new policy would take, and were action probabilities recorded? section 5.3
  • Effective sample size and weights: Is usable evidence only a small fraction of the logged data, or do a few sessions carry most of the weight? section 5.3.2 section 5.4.2
  • Uncertainty and stability: Inspect confidence intervals and see whether the estimate changes substantially when large weights are clipped, a segment is removed, or the time window changes. section 5.3.2

Comparing recommendation policies

A logged movie recommendation policy provides observed rewards for sessions. A new policy’s estimated value is the weighted average of those rewards; the original policy’s performance is their unweighted average. An estimated 8% improvement is less convincing if it depends on a tiny number of heavily weighted sessions. section 5.4.2

Listing 5.2 Calculating Off Policy Evaluation with multiple metrics
def evaluate_policy(self, metric='engagement_score'):
        weights = self.calculate_importance_weights()
        
        if metric == 'click_rate':
            rewards = self.logged_data['clicked'].astype(float)
        elif metric == 'watch_time':
            rewards = self.logged_data['watch_time']
        else:  # engagement_score
            rewards = (self.logged_data['clicked'].astype(float) + self.logged_data['watch_time'] / 100)
        
        new_policy_value = np.sum(weights * rewards) / np.sum(weights) if np.sum(weights) > 0 else 0
        original_policy_value = np.mean(rewards)
        
        return {
            'original_policy_value': original_policy_value,
            'new_policy_value': new_policy_value,
            'improvement': new_policy_value - original_policy_value,
            'relative_improvement': ((new_policy_value / original_policy_value - 1) * 100 if original_policy_value > 0 else 0),
            'effective_sample_size': (np.sum(weights) ** 2) / np.sum(weights ** 2) if np.sum(weights) > 0 else 0
        }
 

Inspect how the evaluation selects rewards and compares weighted and unweighted policy values. section 5.4.2

How AI Model Evaluation covers it

How deep it goes
Defines the metric and develops practical checks for fragile estimates, including weight concentration and sensitivity to analysis choices.
Examples
Compares weighted and unweighted rewards for movie recommendation policies and considers an improvement driven by heavily weighted sessions.
What you can do after
Diagnose fragile off-policy estimates by comparing effective sample size with logged volume and inspecting weight distributions.
How it connects
Connects effective sample size to overlap, inverse propensity weighting, confidence intervals, and importance weights.

Also explained in

Compare how 2 books explain effective sample size →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.