Evaluation measures whether a search model generalizes beyond its training examples. Hold out queries or judgments, train only on the training portion, then rank unseen test queries and calculate a relevance metric such as precision. section 10.5.3 section 12.1.2
A practical evaluation loop is:
- Split data at the query level so judgments from one query do not leak between training and testing. section 10.5.3
- Train the ranking model only on the training judgments. section 10.5.3
- Rank each test query, count relevant results among the top k, and average precision across queries. section 10.5.3
- Compare feature choices and revisit incorrectly ranked results, while treating offline scores as evidence rather than a guarantee of live success. section 10.5.3 section 12.1.2
Listing 10.15 Can our model generalize beyond the training data?
def score_one(features, model):
score = 0.0
for idx, f in enumerate(features):
this_coef = model.coef_[0][idx].item()
score += f * this_coef
return score
def rank(query_judgments, model):
for j in query_judgments:
j.score = score_one(j.features, model)
return sorted(query_judgments, key=lambda j: j.score, reverse=True)
def evaluate_model(test_data, model, k=5):
total_precision = 0
unique_queries = groupby(test_data, lambda j: j.qid)
num_groups = 0
for qid, query_judgments in unique_queries: #1
num_groups += 1
ranked = rank(list(query_judgments), model) #2
total_relevant = len([j for j in ranked[:k]
if j.grade == 1]) #3
total_precision += total_relevant / float(k)
return total_precision / num_groups
evaluation = evaluate_model(test_data, model)
print(evaluation)
The implementation groups judgments by query, scores and sorts them with a trained model, computes top-k precision, and averages the result. section 10.5.3
