liveBook · Manning

Recommendation model

As explained in AI Model Evaluation

A model that selects or ranks items for presentation to users based on predicted interest or utility.

A recommendation model selects or ranks items for a user based on predicted interest or utility. For movie recommendations, it can use a user’s watch history and preferences to produce a personalized list. Evaluation should check not only whether the movies fit the user, but also whether the list is diverse and exposes the breadth of the catalog. section 3.1.1 section 9.6.1

An offline diagnostic can reveal whether recommendations favor blockbusters over indie films, under-recommend films with diverse casts, or repeatedly serve the same genre. Compare recommendation outputs across user segments using movie metadata such as genre, popularity, release year, cast, and language. Historical behavior can help construct realistic profiles; watched or highly rated movies provide labels for comparing exposure with relevance or preference. section 3.1.1

Compare genre exposure by age group

Group recommended movies by age group and genre, then divide each genre count by the total recommendations for that age group. The resulting proportions show how exposure differs across segments. A difference is a reason to investigate, not proof of a problem: check it against user preferences and relevance. section 3.1.1

Listing 3.1 Analyzing Distribution of Genres by Age Group Cohorts
# Calculate the distribution of genres within each age group's recommendations
genre_diversity = analysis_df.groupby(['age_group', 'genre']).size().reset_index(name='count')
genre_pivot = genre_diversity.pivot(index='age_group', columns='genre', values='count').fillna(0)
genre_pivot = genre_pivot.div(genre_pivot.sum(axis=1), axis=0)  # Normalize by row to get proportions
 
genre_pivot
 

The listing groups recommendations by age and genre, pivots the counts, and normalizes each row into proportions. section 3.1.1

An LLM-as-a-judge can assess whether a user’s top-five recommendations are relevant and diverse. An underspecified request to judge the list can produce vague reasoning, inconsistent scores, and unstructured responses that are hard to compare or parse. Define evaluation criteria and an output format rather than relying on a general request for a judgment. section 9.6.1

How AI Model Evaluation covers it

How deep it goes
Works through subgroup exposure analysis and judge-prompt failure modes, without explaining how to build the recommendation model.
Examples
Uses age-group genre distributions and an LLM judgment of top-five movie recommendations.
What you can do after
Evaluate and understand recommendation exposure across user segments and identify weaknesses in an LLM judge prompt.
How it connects
Connects recommendation models to bias, offline evaluation, relevance, diversity, and LLM-as-a-judge.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.