A search model can rank its training examples well without ranking unfamiliar queries well. An evaluation dataset holds back labeled examples so predictions can be checked against answers the model did not train on. For learning to rank, split by query rather than by individual judgment: each held-out query can then be evaluated as a complete result set. section 10.5.3 section 12.1.2
Names for held-out slices vary by workflow. In the question-answering workflow here, training, testing, and validation are three separate slices. The test slice supports development-time evaluation and tuning; the holdout validation slice checks the trained model on examples kept out of both training and testing. This makes the final check more informative than repeatedly measuring performance on familiar examples. section 14.2.4 section 14.4
Build an evaluation dataset around the decision the model must make:
- For a ranker, reserve whole queries and their relevance judgments. Retrain on the remaining queries, rank each held-out result set, and measure relevance near the top—for example, precision at five. section 10.5.3
- For a question-answering reader, start with trustworthy question, context, and answer labels. Automatically generated guesses can form a silver set, but human correction produces the golden set used for splitting. section 14.2 section 14.2.3
- Keep the final holdout separate while developing the model. Evaluate it after training; a worse holdout loss than the earlier evaluation loss is a warning sign of overfitting. section 14.3.4
A held-out question-answering check
A labeled outdoors question-answering dataset is formatted into 125 training examples, 32 testing examples, and 10 validation examples. After fine-tuning, `trainer.evaluate(eval_dataset=tokenized_datasets["validation"])` measures loss on the ten-example holdout. A separate pass compares extracted answers with its labels and gets 7 of 10 correct. The small holdout provides a useful check, not a guarantee of performance on every real-world question. section 14.2.4 section 14.3.4 section 14.4
Listing 10.15 Can our model generalize beyond the training data?
def score_one(features, model):
score = 0.0
for idx, f in enumerate(features):
this_coef = model.coef_[0][idx].item()
score += f * this_coef
return score
def rank(query_judgments, model):
for j in query_judgments:
j.score = score_one(j.features, model)
return sorted(query_judgments, key=lambda j: j.score, reverse=True)
def evaluate_model(test_data, model, k=5):
total_precision = 0
unique_queries = groupby(test_data, lambda j: j.qid)
num_groups = 0
for qid, query_judgments in unique_queries: #1
num_groups += 1
ranked = rank(list(query_judgments), model) #2
total_relevant = len([j for j in ranked[:k]
if j.grade == 1]) #3
total_precision += total_relevant / float(k)
return total_precision / num_groups
evaluation = evaluate_model(test_data, model)
print(evaluation)
Python ranking evaluation groups held-out judgments by query, scores each result set, and averages precision at K. section 10.5.3
