liveBook · Manning

Indexing pipeline

As explained in Context Engineering

An indexing pipeline is a sequence of processing steps that transforms source documents into structures suitable for retrieval.

An indexing pipeline prepares external knowledge for retrieval in a retrieval-augmented generation (RAG) system. It runs offline and asynchronously, ahead of user queries, so it can perform work such as document parsing, chunking, and batch embedding generation without adding that work to the active request path. section 3.2 section 3.2.1

A common dense-retrieval pipeline starts with pointers to source data and proceeds through four stages:

  • Load and preprocess data: extract and normalize text from sources such as repositories, web content, and archives; preserve metadata and anonymize data when required. section 3.2.1
  • Split text into coherent chunks. Fixed-size, semantic, and hierarchical strategies differ in how they preserve contextual relationships. section 3.2.1
  • Convert chunks into embeddings with an embedding model. These vectors support retrieval of conceptually related text rather than only exact keyword matches. section 3.2.1
  • Store embeddings in a vector database, such as ChromaDB or Pinecone, for high-dimensional indexing and nearest-neighbor search. section 3.2.1

The online generation pipeline searches the prepared index at query time, places retrieved context in a prompt, and sends it to the model. Grounding does not guarantee accuracy: source freshness and authority, chunking, ranking, permission filters, and prompt assembly all affect the result. section 3.2 section 3.10.1

A minimal sparse index

A simpler RAG implementation uses scikit-learn’s TF-IDF vectorizer to fit and transform a document corpus into sparse vectors. At query time, the same vectorizer transforms the question; cosine similarity ranks the document vectors, and the top matches supply context for an OpenAI model. This makes the mechanics inspectable without using dense embeddings or a vector database. section 3.10.1

Listing 3.1 Minimal RAG in Python: TF-IDF indexing, similarity-based retrieval, and augmented generation
corpus = [doc["text"] for doc in documents] #1
vectorizer = TfidfVectorizer(stop_words="english")
doc_vectors = vectorizer.fit_transform(corpus) #2

def retrieve(query: str, k: int = 3) -> List[Dict[str, Any]]: #3
    query_vec = vectorizer.transform([query])
    similarities = cosine_similarity(query_vec, doc_vectors)[0]
    top_indices = np.argsort(similarities)[::-1][:k]
    results: List[Dict[str, Any]] = []
    for idx in top_indices:
        results.append({
            "id": documents[idx]["id"],
            "title": documents[idx]["title"],
            "score": float(similarities[idx]),
            "text": documents[idx]["text"],
        })
    return results

def query_model_with_rag(question: str,
                k: int = 3,
                model: str = "gpt-4o-mini") -> str: #4
    retrieved_docs = retrieve(question, k=k)
    context = build_context(retrieved_docs)
    prompt = (
        "You are a helpful assistant ...\n"
        "Documentation:\n"
        f"{context}\n\n"
        f"User question: {question}"
    )
    # Query model (see full source code in GitHub)

Inspect the Python implementation of TF-IDF indexing and similarity-based retrieval. section 3.10.1

How Context Engineering covers it

How deep it goes
Details loading, chunking, embedding, and vector storage, and demonstrates a simpler sparse indexing implementation.
Examples
Illustrates a TF-IDF document index queried with cosine similarity before generation by an OpenAI model.
What you can do after
Builds a minimal RAG pipeline from scratch using TF-IDF indexing and similarity-based retrieval.
How it connects
Connects indexing to chunking, embeddings, vector databases, similarity search, and prompt augmentation.

Also explained in

Compare how 2 books explain indexing pipeline →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.