A diagnostic evaluation examines where a model succeeds or fails, under what conditions, and for which users, items, or tasks. It looks for patterns, biases, and weaknesses that an aggregate performance score can hide. Performance evaluation asks whether a model meets its objective; diagnostics investigate the behavior behind the result. section 2.1.2 section 3.1.2
Diagnostics can reuse performance metrics such as recall, but break them down by meaningful segments or scenarios. An acceptable overall recall, for example, can conceal poor recall for users who prefer foreign-language films. Other diagnostics examine errors, fairness, generalization, coverage, or diversity. section 3.1.2 section 2.1.2
A practical diagnostic evaluation follows a focused process:
- Set a question that can guide a decision. For example, ask whether users with fewer than three prior interactions receive less relevant recommendations. section 3.6.1
- Collect outputs and context suited to the question: recommendation positions, user segments, item metadata, interaction history, and relevance labels where needed. section 3.6.2
- Compare meaningful slices or conditions, using metrics and visualizations that expose the behavior. Check that segment definitions are reliable and that each slice has enough data. section 3.1.1 section 3.6.2
- Investigate the finding, make a targeted change, and rerun the diagnostic. Add important slices and stress tests to the regular evaluation suite. section 3.4.6 section 3.6.5
Movie recommendation exposure
To investigate whether recommendations favor blockbusters or narrow genres, group recommendation outputs by user segment and join them with movie metadata. Compare each segment’s genre proportions, average popularity, or catalog diversity. A heatmap can reveal exposure differences worth investigating; a difference alone does not establish bias, because preferences may also differ. Use viewing history or relevance labels to assess that context. section 3.1.1 section 3.6.1
Listing 3.1 Analyzing Distribution of Genres by Age Group Cohorts
# Calculate the distribution of genres within each age group's recommendations genre_diversity = analysis_df.groupby(['age_group', 'genre']).size().reset_index(name='count') genre_pivot = genre_diversity.pivot(index='age_group', columns='genre', values='count').fillna(0) genre_pivot = genre_pivot.div(genre_pivot.sum(axis=1), axis=0) # Normalize by row to get proportions genre_pivot
The listing groups recommendations by age and genre, then normalizes each age group’s counts into proportions for comparison. section 3.1.1
The same approach applies beyond recommenders. Search diagnostics can compare clean queries with typos or ambiguous phrasing. LLM and agent diagnostics can break down hallucinations by intent or test recovery when a tool call times out. section 3.4.1 section 3.4.5
Detailed logs make these comparisons possible: capture output attributes, ranking positions, user segments, and session context rather than only aggregate scores. Interpret small or unusual slices cautiously, document assumptions, and include underrepresented scenarios. An offline finding is a product hypothesis, not proof of harm or impact; validate it with online experiments, user feedback, or downstream product metrics. section 3.7.1 section 3.6.2 section 3.3
