liveBook · Manning

Indexing pipeline

An indexing pipeline is a sequence of processing steps that transforms source documents into stored representations suitable for retrieval.

How 2 Manning books explain it

Context Engineering

An indexing pipeline is a sequence of processing steps that transforms source documents into structures suitable for retrieval.

How it is used
Prepares external knowledge offline for retrieval during RAG generation.
How deep it goes
Details loading, chunking, embedding, and vector storage, and demonstrates a simpler sparse indexing implementation.
Examples
Illustrates a TF-IDF document index queried with cosine similarity before generation by an OpenAI model.
What you can do after
Builds a minimal RAG pipeline from scratch using TF-IDF indexing and similarity-based retrieval.

Read how Context Engineering explains indexing pipeline →

Retrieval Augmented Generation, The Foundational Ideas

An indexing pipeline is an offline workflow that transforms source documents into stored representations suitable for retrieval.

How it is used
Prepares documents offline so RAG retrieval can supply relevant context to generation at query time.
How deep it goes
Explains chunk-based storage and entity-profile construction, including cost scaling, but not detailed implementation mechanics.
Examples
Uses product manuals, policies, and FAQs, plus LightRAG profiles aggregated from repeated company mentions.
What you can do after
Understand how offline document preparation and entity-profile construction support retrieval at query time.

Read how Retrieval Augmented Generation, The Foundational Ideas explains indexing pipeline →

What to know first

Related concepts

Questions about indexing pipeline

What is indexing pipeline?

An indexing pipeline is a sequence of processing steps that transforms source documents into stored representations suitable for retrieval.

Which Manning books explain indexing pipeline?

Context Engineering and Retrieval Augmented Generation, The Foundational Ideas.

What should I know before learning indexing pipeline?

Manning books that teach indexing pipeline usually introduce similarity search first.

What is indexing pipeline a kind of?

Indexing pipeline is a kind of pipeline. A pipeline is an ordered sequence of connected processing stages through which data or work flows, with each stage passing or transforming its output for subsequent stages.

What is indexing pipeline part of?

Indexing pipeline is part of LightRAG. LightRAG is a graph-enhanced retrieval-augmented generation architecture that maintains separate representations for specific information and conceptual information while profiling graph nodes and edges.

What are the parts of indexing pipeline?

Indexing pipeline includes chunking, data conversion, data ingestion and vector storage.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.