liveBook · Manning

Indexing pipeline

As explained in Retrieval Augmented Generation, The Foundational Ideas

An indexing pipeline is an offline workflow that transforms source documents into stored representations suitable for retrieval.

An indexing pipeline prepares information for retrieval before a user asks a question. In RAG, it runs offline: documents are split into chunks, converted into searchable representations, and stored in a specialized database. The online query pipeline then retrieves and ranks relevant content for generation. section 1.2

LightRAG indexes chunks by extracting entities and accumulating descriptions across the corpus. It stores one LLM-generated profile per entity, which graph retrieval can look up later. Indexing cost scales with unique entities rather than mentions, making it cheaper than GraphRAG in this comparison. section 12.4.3

Customer support and entity profiles

A support system can index product manuals, policy documents, and FAQs so warranty questions can retrieve relevant evidence. In LightRAG, a question about a company can retrieve its profile, aggregated from mentions across the corpus. section 1.2 section 12.4.3

Listing 12.5 Building deduplicated entity profiles.
from typing import Dict, List
import json
 
class LightRAGIndexer:
    """Build a deduplicated entity graph from text chunks."""
 
    def __init__(self, llm, graph: nx.DiGraph):
        """Bind to an LLM client and an empty knowledge graph."""
        self.llm = llm
        self.graph = graph
        self.mentions: Dict[str, List[str]] = {}  #A
 
    def index_chunk(self, chunk_id: str, text: str) -> None:
        """Extract entities and accumulate per-entity context."""
        prompt = (
            "Extract entities and short descriptions from this text. "
            "Return JSON: [{\"entity\": str, "
            "\"type\": str, \"desc\": str}].\n"
            f"Text: {text}"
        )
        items = json.loads(self.llm.complete(prompt))  #B
        for item in items:
            name = item["entity"]
            self.mentions.setdefault(name, []).append(item["desc"])  #C
            if name not in self.graph:
                self.graph.add_node(name, type=item["type"])
 
    def finalize_profiles(self) -> None:
        """Compress each entity's mentions into one profile."""
        for name, descriptions in self.mentions.items():
            joined = "\n".join(descriptions)
            profile_prompt = (
                f"Combine these mentions of {name} into a "
                f"short profile:\n{joined}"
            )
            self.graph.nodes[name]["profile"] = (  #D
                self.llm.complete(profile_prompt))
 

The implementation illustrates extracting entities from chunks and building deduplicated profiles. section 12.4.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains chunk-based storage and entity-profile construction, including cost scaling, but not detailed implementation mechanics.
Examples
Uses product manuals, policies, and FAQs, plus LightRAG profiles aggregated from repeated company mentions.
What you can do after
Understand how offline document preparation and entity-profile construction support retrieval at query time.
How it connects
Pairs document chunking with retrieval and generation; LightRAG connects entity profiles to graph retrieval and vector similarity.

Also explained in

Compare how 2 books explain indexing pipeline →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.