A generation pipeline is the online, synchronous path in retrieval-augmented generation (RAG): it searches external knowledge at query time, adds relevant context to a prompt, and generates a response. Unlike the offline indexing pipeline, its operations run while the user waits and must fit an interactive latency budget. section 3.2
A typical request passes through these stages:
- Optionally transform the query by rewriting, expanding, or decomposing it to improve alignment with the indexed corpus. section 3.2.2
- Retrieve matching chunks from the index. In dense retrieval, the query is embedded with the same model used for indexing and compared with stored vectors. section 3.2.2
- Select the top-K candidates and, where appropriate, rerank them with a more precise model. Top-K balances useful coverage against irrelevant context. section 3.2.2
- Insert the retrieved context into a prompt and send it to the model. A prompt can instruct the model to answer only from the supplied context. section 3.10.1
Minimal sparse retrieval
A Python RAG implementation indexes documents with a TF-IDF vectorizer. At query time, it transforms the question with that same vectorizer, ranks document vectors by cosine similarity, and places the highest-scoring matches in the model prompt. This makes the mechanics inspectable without a dense vector database. section 3.10.1
Listing 3.1 Minimal RAG in Python: TF-IDF indexing, similarity-based retrieval, and augmented generation
corpus = [doc["text"] for doc in documents] #1
vectorizer = TfidfVectorizer(stop_words="english")
doc_vectors = vectorizer.fit_transform(corpus) #2
def retrieve(query: str, k: int = 3) -> List[Dict[str, Any]]: #3
query_vec = vectorizer.transform([query])
similarities = cosine_similarity(query_vec, doc_vectors)[0]
top_indices = np.argsort(similarities)[::-1][:k]
results: List[Dict[str, Any]] = []
for idx in top_indices:
results.append({
"id": documents[idx]["id"],
"title": documents[idx]["title"],
"score": float(similarities[idx]),
"text": documents[idx]["text"],
})
return results
def query_model_with_rag(question: str,
k: int = 3,
model: str = "gpt-4o-mini") -> str: #4
retrieved_docs = retrieve(question, k=k)
context = build_context(retrieved_docs)
prompt = (
"You are a helpful assistant ...\n"
"Documentation:\n"
f"{context}\n\n"
f"User question: {question}"
)
# Query model (see full source code in GitHub)
Inspect the Python listing for vectorization, similarity scoring, top-K selection, and augmented generation. section 3.10.1
Grounding does not guarantee a reliable answer. Source freshness and authority, chunking, ranking, permission filters, and prompt assembly can each affect whether the model receives appropriate evidence. Query transformation, retrieval, reranking, and inference also contribute to request latency. section 3.2 section 3.2.2
