A foundational model, also called a foundation model, is pretrained on broad data for use across many tasks. In search, it provides a starting point for interpreting queries and documents, producing embeddings, or generating responses. A large language model (LLM) is a text-focused type of foundation model; other models can work with images, audio, or multiple data types. section 15.1 section 1.1
A useful foundation model has several properties:
- It learns general patterns from substantial data before a particular search task is chosen. Its trained weights can then be deployed or fine-tuned. section 15.1.1
- It can be adapted with task-specific or domain-specific data. For semantic search, fine-tuning on labeled similar and dissimilar passages can improve text-similarity matching over an untuned model. section 1.3.4 section 13.4.2
- It need not start with text: image embeddings support image-to-image search, while a model trained on text and images can support searching images with text queries. section 15.1
Reusing a pretrained model avoids the substantial computing cost of training one from scratch, but its general knowledge is not a dependable store of facts. Generated answers can invent or confuse details. For factual search responses, retrieve relevant documents first and give the model that context to summarize with citations; check the resulting claims against their sources. section 13.3.2 section 15.2.2
Bootstrapping question-answering labels
Suppose a retriever supplies question-and-document-context pairs. Run them through the Hugging Face question-answering pipeline with `deepset/roberta-base-squad2`, a RoBERTa-based model already fine-tuned for question answering. Treat its outputs as guesses in a silver set, not verified answers. Review and correct them to produce a golden set for further work. section 14.2.2
Listing 14.6 Generating answers given question/context pairs
from transformers import pipeline #1
import torch
import tqdm #2
def get_processor_device(): #3
return 0 if torch.cuda.is_available() else -1 #3
def answer_questions(contexts, k=10):
nlp = pipeline("question-answering", model=model_name, #4
tokenizer=model_name, device=device) #4
guesses = []
for _, row in tqdm.tqdm(contexts[0:k].iterrows(), total=k): #5
result = nlp({"question": row["question"], #6
"context": row["context"]}) #6
guesses.append(result)
return guesses
model_name = "deepset/roberta-base-squad2"
device = get_processor_device() #7
guesses = answer_questions(contexts, k=len(contexts))
display_guesses(guesses)
The Python listing shows the model and tokenizer passed to a question-answering pipeline, which generates guesses from question/context pairs and selects a CPU or GPU device. section 14.2.2
