liveBook · Manning

Generator

As explained in Retrieval Augmented Generation, The Foundational Ideas

A generator is a model component that produces a natural-language answer conditioned on retrieved documents or passages.

A generator turns a question and retrieved passages into a natural-language answer. Its job is synthesis, not search: the retriever finds relevant evidence, and the generator combines it into a response. This can answer questions requiring multiple facts rather than copying a single text span. section 4.0 section 4.6

In canonical RAG, the generator is a pretrained BART sequence-to-sequence model, paired with a pretrained dense passage retriever. Fine-tuning updates the generator and the retriever’s query encoder while leaving the document encoder and its index fixed. The generator learns from question–answer pairs with retrieved passages in its input, encouraging it to use those passages and refrain from unsupported answers. section 4.1 section 4.4

A simpler, naive RAG pipeline instead places retrieved chunks and the question in a prompt to an off-the-shelf chat model. It can work without fine-tuning, but asking a model to rely on context is less reliable than training it to do so. Neither approach can supply evidence that retrieval failed to find. section 4.4 section 4.6

Answer only from retrieved context

For a question about a technical paper, retrieve three document chunks, join their text, and send them with the question to a chat model. The prompt says to answer only from the context and to say when the answer is absent. One example returns the paper’s two RAG formulations; another declines to name its retriever because the retrieved pages do not contain that answer. Returning page numbers helps inspect what the generator received. section 4.6

Listing 4.3 Retrieving, then generating from what was retrieved.
from langchain_core.prompts import ChatPromptTemplate 
from providers import get_chat_model
 
llm = get_chat_model(temperature=0)  #A
prompt = ChatPromptTemplate.from_template(
  "Answer from the context only. If the context does not " 
  "contain the answer, say so.\n\n{context}\n\n" 
  "Question: {question}"
)  #B
chain = prompt | llm  #C
 
def answer(question, k=3): 
  docs = index.similarity_search(question, k=k)  #D
  context = "\n\n".join(d.page_content for d in docs) 
  reply = chain.invoke({"context": context, "question": question}) 
  pages = [d.metadata["page"] for d in docs]  #E
  return reply.content, pages

The Python listing shows the generator boundary: similarity search supplies passages, a prompt supplies the grounding instruction, and the chat model produces the answer. section 4.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the modular architecture and supervised fine-tuning objective, then implements prompted generation without fine-tuning the example model.
Examples
Uses Python paper Q&A that retrieves three chunks, answers from context, reports pages, and declines an unsupported answer.
What it assumes
Assumes enough Python familiarity to follow an example while walking through chunking, indexing, retrieval, prompting, and generation.
What you can do after
Build it from scratch as a basic retrieve-then-generate pipeline, or evaluate whether a generated answer is supported by retrieved passages.
How it connects
Connects generation to dense passage retrieval, query encoders, document indexes, sequence-to-sequence models, and grounding in retrieved context.

Also explained in

Compare how 13 books explain generator →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.