A model’s performance is not one universal number. Choose evaluation metrics for the business goal before building the model, then use a small set that measures different aspects of its output. Those numbers make validation results easier to discuss, but only if they describe what users need the model to do. section 9.3.3
For a movie recommender, three metrics answer different questions:
- Root mean squared error (RMSE) measures predicted-rating error; lower is better. section 9.3.3
- Precision measures how many recommended items are relevant; higher is better. section 9.3.3
- Recall measures how many relevant items appear among the recommendations; higher is better. section 9.3.3
Calculate these metrics on validation data and retain them alongside model versions, hyperparameters, and validation examples. An experiment tracker such as MLflow helps teams compare runs and investigate later anomalies. Inspect sample outputs manually as well: early debugging can reveal behavior that the current metrics do not represent. Revisit the metrics as the model and its requirements evolve; the recommender’s three measures are rudimentary examples, not a ready-made production scorecard. section 9.3 section 9.3.3 section 9.4
Validate top-50 recommendations
A validation function ranks a user’s recommendations, uses a rating threshold to identify relevance, and computes precision and recall over the top 50. It also computes RMSE for predicted ratings. Keeping these measures separate helps expose a model that predicts ratings well but produces a less useful recommendation list. section 9.3.3
Listing 9.12 Model evaluation code
def validate_model(
recommendation_model,
top_k=50,
threshold=3,
val_batch_size=32):
from collections import defaultdict #1
import torch
from sklearn.metrics import mean_squared_error
def calculate_precision_recall(user_ratings, k, threshold):
user_ratings.sort(key=lambda x: x[0], reverse=True)
n_rel = sum(x >= threshold for _, x in user_ratings)
n_rec_k = sum(x >= threshold for x, _ in user_ratings[:k])
n_rel_and_rec_k = sum(
(x >= threshold) and (y >= threshold)
for y, x in user_ratings[:k]
)
precision = n_rel_and_rec_k / n_rec_k if n_rec_k != 0 else 1
recall = n_rel_and_rec_k / n_rel if n_rel != 0 else 1
return precision, recall
user_ratings_comparison = defaultdict(list)
dataset_map = get_datasets_local(split=["val"])
val_dataloader = DataLoader(
dataset_map["val"],
batch_size=val_batch_size,
shuffle=True
)
y_pred = []
y_true = []
recommendation_model.eval()
with torch.no_grad():
for users, movies, ratings in val_dataloader:
output = recommendation_model(users, movies)
y_pred.append(output.sum().item() / len(users))
y_true.append(ratings.sum().item() / len(users))
for user, pred, true in zip(users, output, ratings):
user_ratings_comparison[user.item()].append(
(pred[0].item(), true.item())
)
user_precisions = dict()
user_based_recalls = dict()
k = top_k
for user_id, user_ratings in user_ratings_comparison.items():
precision, recall = calculate_precision_recall(
user_ratings, k, threshold)
user_precisions[user_id] = precision
user_based_recalls[user_id] = recall
average_precision = sum(prec for prec in user_precisions.values())/len(
user_precisions
)
average_recall = sum(rec for rec in user_based_recalls.values())/len(
user_based_recalls
)
rms = mean_squared_error(y_true, y_pred, squared=False)
print(f"precision_{k}: {average_precision:.4f}")
print(f"recall_{k}: {average_recall:.4f}")
print(f"rms: {rms:.4f}")
The evaluation listing shows how the recommendation threshold and top-k cutoff become inputs to repeatable validation code. section 9.3.3
After deployment, distinguish model quality from service health. A model can score well in validation yet perform worse on live data. Monitor ML-specific measures alongside CPU, memory, request latency, failures, and load; use thresholds and alerts to prompt investigation or another pipeline run. Prometheus can scrape service metrics for Grafana dashboards, while reports built with Evidently can compare reference and current data or display regression errors. Neither a healthy API nor a drift score alone proves that predictions remain useful. section 1.1.2 section 3.5 section 6.2.1 section 11.1.1
For an LLM application, accuracy and latency leave important gaps: an answer can be technically correct but unhelpful, or fail through hallucination or unsafe output. Match evaluation to the behavior that matters, and trace retrieval and context assembly when a RAG answer goes wrong. section 12.4.4 section 12.4.4
