Off-policy evaluation (OPE) estimates how a target policy might perform using data collected by a different behavior policy. It helps teams compare a new recommendation, ranking, or decision strategy with a production policy without exposing users to the change. The result is an estimate under assumptions, not a replay of an alternative reality. section 5.11 section 5.4.2
The quantity of interest is policy value: the expected performance of a policy. Unlike a standard offline evaluation against known labels, OPE estimates outcomes for decisions that were not necessarily taken. section 5.4
A practical OPE workflow starts with the decision and its logs:
- Define the target policy and the outcome that matters, such as engagement, revenue, click-through rate, or viewing time. section 5.4.2
- Record the decision context, candidate actions, selected action, behavior-policy probabilities, policy version, and observed outcome. Calculate the target policy’s action probabilities for the same context and candidates. section 5.11 section 5.4.2
- Estimate target-policy value from logged outcomes. Inverse propensity weighting adjusts outcomes according to the relative probability of the logged action under the target and behavior policies. section 5.11
- Compare the estimate with the existing policy, then inspect uncertainty, weights, effective sample size, and segment-level results before using it for a product decision. section 5.11 section 5.4.2
Evaluating a movie recommendation policy
Suppose logged recommendations favor popularity, while a proposed policy favors highly rated movies and boosts scores for a user’s preferred genres. Apply the proposed policy to the available movies in each logged context, calculate its action probabilities, and use importance weights to estimate click rate or watch time. Compare those estimates with the original policy and check whether gains differ across user segments. section 5.4.2 section 5.8.3
Listing 5.1 Calculating New Policy Probabilities
def new_policy_probabilities(self, user_id, available_movie_ids):
"""Quality-focused policy (vs popularity-based logged policy)"""
user_prefs = self.users_catalog.iloc[user_id]['preferred_genres']
available_movies = self.movies_catalog[self.movies_catalog['movie_id'].isin(available_movie_ids)]
scores = []
for _, movie in available_movies.iterrows():
score = movie['rating'] # Focus on quality
if movie['genre'] in user_prefs:
score *= 2.0 # Strong preference boost
scores.append(score)
scores = np.array(scores)
if len(scores) == 0:
return np.array([])
exp_scores = np.exp(scores - np.max(scores))
return exp_scores / np.sum(exp_scores)
The implementation turns movie ratings and genre preferences into target-policy probabilities for the available movies. section 5.4.2
Listing 5.2 Calculating Off Policy Evaluation with multiple metrics
def evaluate_policy(self, metric='engagement_score'):
weights = self.calculate_importance_weights()
if metric == 'click_rate':
rewards = self.logged_data['clicked'].astype(float)
elif metric == 'watch_time':
rewards = self.logged_data['watch_time']
else: # engagement_score
rewards = (self.logged_data['clicked'].astype(float) + self.logged_data['watch_time'] / 100)
new_policy_value = np.sum(weights * rewards) / np.sum(weights) if np.sum(weights) > 0 else 0
original_policy_value = np.mean(rewards)
return {
'original_policy_value': original_policy_value,
'new_policy_value': new_policy_value,
'improvement': new_policy_value - original_policy_value,
'relative_improvement': ((new_policy_value / original_policy_value - 1) * 100 if original_policy_value > 0 else 0),
'effective_sample_size': (np.sum(weights) ** 2) / np.sum(weights ** 2) if np.sum(weights) > 0 else 0
}
The evaluation implementation compares a weighted target-policy estimate with the logged-policy average for several reward choices. section 5.4.2
Trust depends on accurate propensities and sufficient overlap: the logs must contain enough relevant actions for the target policy to be evaluated. Missing candidate actions, probabilities, or context can make estimates skewed or invalid; unstable weights can also make them unreliable. Exploration, such as a propensity-logged bandit traffic slice, can help preserve information about alternatives. Logging must also balance evaluation needs against latency, storage, privacy, and governance. section 5.11 section 5.7.1 section 8.2.1
