liveBook · Manning

Text pipeline

As explained in Retrieval Augmented Generation, The Foundational Ideas

A text pipeline is a retrieval workflow that represents and searches documents using extracted textual content.

A text pipeline retrieves documents through extracted textual content: extract text during indexing, embed or otherwise index that representation, and retrieve passages for generation. Its production advantage is a stable, existing index; its limitation is losing information that extraction cannot represent, especially page layout and visual structure. section 15.6.5

Use the pipeline as a baseline before adopting visual or late-interaction retrieval:

  • Keep the existing index when the generator can work from retrieved text. section 15.6.5
  • Add a reranker when first-stage text retrieval needs stronger ordering; benchmark the gain rather than assuming visual retrieval is superior. section 15.6.5
  • Combine text and visual results when documents contain complementary textual and rendered-page evidence. section 15.6.5

Extracting a page for indexing

A minimal extraction step opens a PDF page with PyMuPDF, calls get_text(), and normalizes whitespace before indexing the resulting text.

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Contrasts indexing and reranking tradeoffs but stops short of specifying a complete extraction or ranking implementation.
Examples
Uses PDF text extraction, visual page retrieval, reranking results, and hybrid top-five page and passage retrieval.
What it assumes
Assumes documents can be text-extracted and indexed, while leaving extraction quality and ranking implementation unspecified.
What you can do after
Evaluate retrieval architectures by comparing text baselines, rerankers, visual retrievers, hybrid results, and storage costs.
How it connects
Connects document retrieval with embeddings, reranking, visual retrieval, late interaction, and hybrid retrieval.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.