An indexing pipeline prepares external knowledge for retrieval in a retrieval-augmented generation (RAG) system. It runs offline and asynchronously, ahead of user queries, so it can perform work such as document parsing, chunking, and batch embedding generation without adding that work to the active request path. section 3.2 section 3.2.1
A common dense-retrieval pipeline starts with pointers to source data and proceeds through four stages:
- Load and preprocess data: extract and normalize text from sources such as repositories, web content, and archives; preserve metadata and anonymize data when required. section 3.2.1
- Split text into coherent chunks. Fixed-size, semantic, and hierarchical strategies differ in how they preserve contextual relationships. section 3.2.1
- Convert chunks into embeddings with an embedding model. These vectors support retrieval of conceptually related text rather than only exact keyword matches. section 3.2.1
- Store embeddings in a vector database, such as ChromaDB or Pinecone, for high-dimensional indexing and nearest-neighbor search. section 3.2.1
The online generation pipeline searches the prepared index at query time, places retrieved context in a prompt, and sends it to the model. Grounding does not guarantee accuracy: source freshness and authority, chunking, ranking, permission filters, and prompt assembly all affect the result. section 3.2 section 3.10.1
A minimal sparse index
A simpler RAG implementation uses scikit-learn’s TF-IDF vectorizer to fit and transform a document corpus into sparse vectors. At query time, the same vectorizer transforms the question; cosine similarity ranks the document vectors, and the top matches supply context for an OpenAI model. This makes the mechanics inspectable without using dense embeddings or a vector database. section 3.10.1
Listing 3.1 Minimal RAG in Python: TF-IDF indexing, similarity-based retrieval, and augmented generation
corpus = [doc["text"] for doc in documents] #1
vectorizer = TfidfVectorizer(stop_words="english")
doc_vectors = vectorizer.fit_transform(corpus) #2
def retrieve(query: str, k: int = 3) -> List[Dict[str, Any]]: #3
query_vec = vectorizer.transform([query])
similarities = cosine_similarity(query_vec, doc_vectors)[0]
top_indices = np.argsort(similarities)[::-1][:k]
results: List[Dict[str, Any]] = []
for idx in top_indices:
results.append({
"id": documents[idx]["id"],
"title": documents[idx]["title"],
"score": float(similarities[idx]),
"text": documents[idx]["text"],
})
return results
def query_model_with_rag(question: str,
k: int = 3,
model: str = "gpt-4o-mini") -> str: #4
retrieved_docs = retrieve(question, k=k)
context = build_context(retrieved_docs)
prompt = (
"You are a helpful assistant ...\n"
"Documentation:\n"
f"{context}\n\n"
f"User question: {question}"
)
# Query model (see full source code in GitHub)
Inspect the Python implementation of TF-IDF indexing and similarity-based retrieval. section 3.10.1

