Relevance is how well a result addresses a user’s query, interests, or task. In a ranked movie recommendation list, more relevant movies should ideally appear near the top so users can find something they want to watch quickly. For a generated answer, relevance is a separate evaluation dimension from groundedness. section 2.2.3 section 1.3.4
Choose a measure that matches the product question. Ranking and retrieval evaluations can use Precision@K or NDCG, while generated answers can be assessed with a relevance rubric. An LLM-as-a-judge is a scoring method, not a metric itself; its relevance judgments are evaluation signals that need validation and calibration. section 1.3.4 section 2.2.3
Offline evaluation of movie recommendations
Train a genre-based model on historical interactions, generate ranked lists for users in a holdout dataset, and compare recommendations with movies they watched, clicked, rated, or saved. These interactions serve as proxy relevance labels, not perfect preferences: a user may click a movie because it was prominently placed. Add diversity or coverage metrics to check whether relevant-looking lists are too narrow or repetitive. section 2.2.3
Listing 2.1 Compute precision
def calculate_precision_at_k(user_id, test_data, recommendations, k=10):
"""Calculate precision@k for a single user."""
# Get highly rated movies the user interacted with in the test set
actual_movies = set(
test_data[
(test_data["user_id"] == user_id) &
(test_data["rating"] >= 4)
]["movie_id"]
)
if not actual_movies:
return None # No relevant test data for this user
# Get top-k recommended movies
recommended_movies = [movie_id for movie_id, _ in recommendations[:k]]
# Calculate precision: number of hits divided by k
hits = len(set(recommended_movies) & actual_movies)
precision = hits / min(k, len(recommended_movies)) if recommended_movies else 0
return precision
This implementation calculates a user’s Precision@K using movies rated at least 4 in the test data as relevant items. It returns no score when the user has no relevant test data. section 2.2.3
A structured LLM judge can rate a user’s top five recommendations for relevance, diversity, and discovery separately. Give it the viewing history, define each criterion, request a short justification and a 1–5 score, and require JSON output. Check stability across reruns, agreement between models, or correlation with human ratings; gold-standard examples can help calibrate judgments. section 9.6.1
Use a fixed dataset and consistent metric definitions to compare model versions, then inspect user and item segments for problems the aggregate score may hide. An offline relevance improvement alone does not establish a better live product experience; diagnostic evaluations and A/B testing address different parts of that question. section 1.4
















