liveBook · Manning

Policy

As explained in AI Model Evaluation

A decision rule that maps a context to an action or a probability distribution over actions.

A policy is a decision rule that takes context and chooses an action, or assigns probabilities to possible actions. It can be a recommendation algorithm, ranking model, tool-routing strategy, or business rule. Policy value is its expected performance on a chosen outcome, such as click-through rate or watch time. section 5.4.1

Before exposing users to a new policy, off-policy evaluation estimates its value using decisions made by an existing policy. This can help compare a candidate with a baseline, but the result is an estimate, not a replay of what would have happened. section 5.11 section 5.4.2

A practical evaluation follows the decision from logging to comparison:

  • Log the decision context, candidate set, selected action, its probability, policy version, and observed outcome. Capture factors that mattered at decision time. section 5.3 section 5.2.2
  • Replay each logged context through the candidate policy and calculate its probability of taking the action actually logged. Weight the observed outcome by the candidate probability divided by the logged policy’s probability. section 5.4.1
  • Aggregate weighted outcomes for the chosen metric and compare the estimate with the baseline. Inspect confidence intervals, effective sample size, weight distributions, and segment-level results. section 5.8.1 section 5.11

Reliable estimates require accurate action probabilities, sufficient overlap between policies, and relevant logged context. A large dataset is not enough if the old policy rarely explored actions the new policy favors. Unstable weights or estimates that swing across segments or time windows are warning signs. Use the estimate to decide whether a policy warrants an online A/B test, not as proof of user impact. section 5.2.2 section 5.3.2

Evaluating a movie recommendation policy

A candidate policy can score available movies by rating, boost scores for preferred genres, and convert the scores into action probabilities. Logged recommendations, alternatives, session context, and outcomes support a weighted comparison with the original policy. In one illustrative outcome, the candidate estimated 15% higher average viewing time and 10% higher average ratings, with a stronger viewing-time increase on weekends. section 5.4.2 section 5.8.1 section 5.8.3

Listing 5.1 Calculating New Policy Probabilities
def new_policy_probabilities(self, user_id, available_movie_ids):
        """Quality-focused policy (vs popularity-based logged policy)"""
        user_prefs = self.users_catalog.iloc[user_id]['preferred_genres']
        available_movies = self.movies_catalog[self.movies_catalog['movie_id'].isin(available_movie_ids)]
        
        scores = []
        for _, movie in available_movies.iterrows():
            score = movie['rating']  # Focus on quality
            if movie['genre'] in user_prefs:
                score *= 2.0  # Strong preference boost
            scores.append(score)
        
        scores = np.array(scores)
        if len(scores) == 0:
            return np.array([])
        
        exp_scores = np.exp(scores - np.max(scores))
        return exp_scores / np.sum(exp_scores)
 

Inspect how movie ratings and genre preferences become probabilities over the available candidates. section 5.4.2

How AI Model Evaluation covers it

How deep it goes
Defines policy and policy value, then walks through logging, probability weighting, comparison, and reliability checks.
Examples
Illustrates a movie policy that scores ratings and preferred genres, alongside a weighted recommendation comparison.
What you can do after
Evaluate a candidate policy against a baseline using logged outcomes while recognizing when the estimate is unreliable.
How it connects
Connects policy to off-policy evaluation, inverse propensity weighting, overlap, and A/B testing.

Also explained in

Compare how 21 books explain policy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.