A language model can produce a plausible answer without having evidence for it. Retrieval-augmented generation (RAG) gives it relevant text to work from: search a document collection, place the retrieved passages in the prompt, then generate an answer based on those passages. For question answering, this turns an open-ended generation task into something closer to search and paraphrasing. section 10.1 section 12.4
A practical RAG pipeline has three parts:
- Keep a searchable corpus on the subjects users will ask about; full-text search or semantic search can retrieve candidate passages. section 12.4
- Pass the question and retrieved documents to a generator. A prompt can explicitly require an answer based on those documents and an “I don't know” response when they do not suffice. section 10.3.11
- Connect retrieval to generation so each question triggers a search before the model answers. section 10.3.11
Listing 10.23 Creating a Haystack PromptNode for RAG
>>> from haystack.nodes import PromptNode, PromptTemplate, AnswerParser
rag_prompt = PromptTemplate(
prompt="""Synthesize a comprehensive answer from the
following text for the given question.
Provide a clear response that summarizes the
key points and information presented in the text.
If you can't answer the question based on
provided sources, say 'I don't know'.
\n\n Related text: {join(documents)}
\n\n Question: {query}
\n\n Answer:""",
)
>>> prompt_node = PromptNode(model_name_or_path="google/flan-t5-base", default_prompt_template=rag_prompt)
This Haystack prompt template inserts retrieved documents and the user's question, then tells FLAN-T5 to say “I don't know” when the sources cannot answer. section 10.3.11
>>> rag_pipe = Pipeline() >>> rag_pipe.add_node(component=retriever, name="Retriever", inputs=["Query"]) >>> rag_pipe.add_node(component=prompt_node, name="prompt_node", inputs=["Retriever"])
The pipeline sends a query through the retriever before passing its output to the prompt node. section 10.3.11
Answer only when the documents help
With five retrieved documents, the pipeline answers a question comparing LSTM networks and RNNs. Asked instead where the Louvre Museum is, it returns “I don't know” rather than supplying an answer from the model's general knowledge. The distinction matters when answers must come from a chosen corpus. section 10.3.11
RAG can bring private documents into a question-answering system without putting them in a model's training data. A self-hosted document store and locally hosted generator are options when privacy matters. Its usefulness still depends on having a corpus that contains the needed information and retrieving the right text; the instruction to decline unsupported questions is part of the design, not a substitute for checking answers. section 12.4 section 10.3.11
