A recommendation model selects or ranks items for a user based on predicted interest or utility. For movie recommendations, it can use a user’s watch history and preferences to produce a personalized list. Evaluation should check not only whether the movies fit the user, but also whether the list is diverse and exposes the breadth of the catalog. section 3.1.1 section 9.6.1
An offline diagnostic can reveal whether recommendations favor blockbusters over indie films, under-recommend films with diverse casts, or repeatedly serve the same genre. Compare recommendation outputs across user segments using movie metadata such as genre, popularity, release year, cast, and language. Historical behavior can help construct realistic profiles; watched or highly rated movies provide labels for comparing exposure with relevance or preference. section 3.1.1
Compare genre exposure by age group
Group recommended movies by age group and genre, then divide each genre count by the total recommendations for that age group. The resulting proportions show how exposure differs across segments. A difference is a reason to investigate, not proof of a problem: check it against user preferences and relevance. section 3.1.1
Listing 3.1 Analyzing Distribution of Genres by Age Group Cohorts
# Calculate the distribution of genres within each age group's recommendations genre_diversity = analysis_df.groupby(['age_group', 'genre']).size().reset_index(name='count') genre_pivot = genre_diversity.pivot(index='age_group', columns='genre', values='count').fillna(0) genre_pivot = genre_pivot.div(genre_pivot.sum(axis=1), axis=0) # Normalize by row to get proportions genre_pivot
The listing groups recommendations by age and genre, pivots the counts, and normalizes each row into proportions. section 3.1.1
An LLM-as-a-judge can assess whether a user’s top-five recommendations are relevant and diverse. An underspecified request to judge the list can produce vague reasoning, inconsistent scores, and unstructured responses that are hard to compare or parse. Define evaluation criteria and an output format rather than relying on a general request for a judgment. section 9.6.1
