A training dataset is the examples a model uses to learn. In search, those examples might describe how relevant a document is to a query; in question answering, they might pair a question and passage with an answer. The labels matter as much as the examples: a model trained on incomplete or misleading judgments can learn the wrong ranking. section 11.0 section 14.2 section 12.2.4
For learning to rank, build the dataset as a pipeline:
- Collect search sessions with the query, displayed rank, document, and whether the result was clicked. section 12.1.1
- Convert clicks into query–document judgments. A click model estimates whether a result was examined; clicked and examined counts yield relevance grades. A beta prior tempers grades when evidence is sparse. section 12.1.1
- Log search features for the judged documents, transform the observations into a pairwise training set, and train the ranking model. section 11.4 section 10.5
From sessions to judgments
For the query “blue ray,” one document has 42 clicks and 42 estimated examinations. Its raw grade is 1.0, but its prior-adjusted beta grade is about 0.85. The adjustment prevents a limited observation from receiving the same certainty as much more extensive evidence. In Python, `generate_training_data(sessions, prior_weight=10, prior_grade=0.2)` produces these query–document judgments. section 12.1.1
Reserve examples for evaluation instead of fitting on every judgment. One workflow splits judgments into train and test sets, trains and uploads the model using the train set, then evaluates it on the excluded test set. Good offline results are not a guarantee: an A/B test can reveal that the judgments reflected position bias, sparse-click uncertainty, or other behavior besides relevance. section 12.1.2 section 12.2.4
Coverage is another constraint. If the judgments for “transformers dvd” contain name matches but no promoted items, the model has no training evidence for those missing combinations. Inspect feature coverage and gather evidence for blind spots rather than assuming that adding features alone will help. More features require relevant and irrelevant examples across the combinations the model must learn. section 12.3.1 section 10.7
For question answering, start with suitable question–answer content. Automatically generated labels form a silver set; manual correction produces a more trustworthy golden set. Format the labeled examples for the training pipeline, then separate training, testing, and holdout validation slices to check whether the fine-tuned model generalizes beyond examples it saw during training. section 14.2 section 14.2.4
Listing 14.7 Transforming a golden dataset into SQuAD formatting
from datasets import Dataset, DatasetDict
def get_training_data(filename):
golden_answers = pandas.read_csv(filename)
golden_answers = golden_answers[golden_answers["class"] != None]
qa_data = []
for _, row in golden_answers.iterrows():
answers = row["gold"].split("|")
starts = [row["context"].find(a) for a in answers]
missing = -1 in starts
if not missing:
row["title"] = row["question"]
row["answers"] = {"text": answers, "answer_start": starts}
qa_data.append(row)
columns = ["id", "url", "title", "question", "context", "answers"]
df = pandas.DataFrame(qa_data, columns=columns) \ #1
.sample(frac=1, random_state=0) #1
train_split = int(len(df) * 0.75) #2
eval_split = (int((len(df) - train_split) / 1.25) + #3
train_split - 1) #3
train_dataset = Dataset.from_pandas(df[:train_split])
test_dataset = Dataset.from_pandas(df[train_split:eval_split])
validation_dataset = Dataset.from_pandas(df[eval_split:]) #4
return DatasetDict({"train": train_dataset, #5
"test": test_dataset, #5
"validation": validation_dataset}) #5
datadict = get_training_data("data/outdoors/outdoors_golden_answers.csv")
model_path = "data/question-answering/question-answering-training-set"
datadict.save_to_disk(model_path)
The Python listing shows a concrete conversion from labeled CSV rows to a Hugging Face dataset dictionary with separate training, testing, and validation slices. section 14.2.4
