liveBook · Manning

Off-policy evaluation

As explained in AI Model Evaluation

Off-policy evaluation estimates the performance of a target policy using data collected under a different behavior policy.

Off-policy evaluation (OPE) estimates how a target policy might perform using data collected by a different behavior policy. It helps teams compare a new recommendation, ranking, or decision strategy with a production policy without exposing users to the change. The result is an estimate under assumptions, not a replay of an alternative reality. section 5.11 section 5.4.2

The quantity of interest is policy value: the expected performance of a policy. Unlike a standard offline evaluation against known labels, OPE estimates outcomes for decisions that were not necessarily taken. section 5.4

A practical OPE workflow starts with the decision and its logs:

  • Define the target policy and the outcome that matters, such as engagement, revenue, click-through rate, or viewing time. section 5.4.2
  • Record the decision context, candidate actions, selected action, behavior-policy probabilities, policy version, and observed outcome. Calculate the target policy’s action probabilities for the same context and candidates. section 5.11 section 5.4.2
  • Estimate target-policy value from logged outcomes. Inverse propensity weighting adjusts outcomes according to the relative probability of the logged action under the target and behavior policies. section 5.11
  • Compare the estimate with the existing policy, then inspect uncertainty, weights, effective sample size, and segment-level results before using it for a product decision. section 5.11 section 5.4.2

Evaluating a movie recommendation policy

Suppose logged recommendations favor popularity, while a proposed policy favors highly rated movies and boosts scores for a user’s preferred genres. Apply the proposed policy to the available movies in each logged context, calculate its action probabilities, and use importance weights to estimate click rate or watch time. Compare those estimates with the original policy and check whether gains differ across user segments. section 5.4.2 section 5.8.3

Listing 5.1 Calculating New Policy Probabilities
def new_policy_probabilities(self, user_id, available_movie_ids):
        """Quality-focused policy (vs popularity-based logged policy)"""
        user_prefs = self.users_catalog.iloc[user_id]['preferred_genres']
        available_movies = self.movies_catalog[self.movies_catalog['movie_id'].isin(available_movie_ids)]
        
        scores = []
        for _, movie in available_movies.iterrows():
            score = movie['rating']  # Focus on quality
            if movie['genre'] in user_prefs:
                score *= 2.0  # Strong preference boost
            scores.append(score)
        
        scores = np.array(scores)
        if len(scores) == 0:
            return np.array([])
        
        exp_scores = np.exp(scores - np.max(scores))
        return exp_scores / np.sum(exp_scores)
 

The implementation turns movie ratings and genre preferences into target-policy probabilities for the available movies. section 5.4.2

Listing 5.2 Calculating Off Policy Evaluation with multiple metrics
def evaluate_policy(self, metric='engagement_score'):
        weights = self.calculate_importance_weights()
        
        if metric == 'click_rate':
            rewards = self.logged_data['clicked'].astype(float)
        elif metric == 'watch_time':
            rewards = self.logged_data['watch_time']
        else:  # engagement_score
            rewards = (self.logged_data['clicked'].astype(float) + self.logged_data['watch_time'] / 100)
        
        new_policy_value = np.sum(weights * rewards) / np.sum(weights) if np.sum(weights) > 0 else 0
        original_policy_value = np.mean(rewards)
        
        return {
            'original_policy_value': original_policy_value,
            'new_policy_value': new_policy_value,
            'improvement': new_policy_value - original_policy_value,
            'relative_improvement': ((new_policy_value / original_policy_value - 1) * 100 if original_policy_value > 0 else 0),
            'effective_sample_size': (np.sum(weights) ** 2) / np.sum(weights ** 2) if np.sum(weights) > 0 else 0
        }
 

The evaluation implementation compares a weighted target-policy estimate with the logged-policy average for several reward choices. section 5.4.2

Trust depends on accurate propensities and sufficient overlap: the logs must contain enough relevant actions for the target policy to be evaluated. Missing candidate actions, probabilities, or context can make estimates skewed or invalid; unstable weights can also make them unreliable. Exploration, such as a propensity-logged bandit traffic slice, can help preserve information about alternatives. Logging must also balance evaluation needs against latency, storage, privacy, and governance. section 5.11 section 5.7.1 section 8.2.1

How AI Model Evaluation covers it

How deep it goes
Develops a logging and evaluation workflow with probability and weighted-value implementations, then identifies overlap and stability checks.
Examples
Uses a movie recommender that contrasts a quality-focused target policy with popularity-based logs and evaluates clicks and watch time.
What you can do after
Evaluate potential policy performance using logged outcomes, action probabilities, and reliability checks before making a deployment decision.
How it connects
Connects policy value with counterfactual logging, inverse propensity weighting, and exploration through multi-armed bandits.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.