A text pipeline retrieves documents through extracted textual content: extract text during indexing, embed or otherwise index that representation, and retrieve passages for generation. Its production advantage is a stable, existing index; its limitation is losing information that extraction cannot represent, especially page layout and visual structure. section 15.6.5
Use the pipeline as a baseline before adopting visual or late-interaction retrieval:
- Keep the existing index when the generator can work from retrieved text. section 15.6.5
- Add a reranker when first-stage text retrieval needs stronger ordering; benchmark the gain rather than assuming visual retrieval is superior. section 15.6.5
- Combine text and visual results when documents contain complementary textual and rendered-page evidence. section 15.6.5
Extracting a page for indexing
A minimal extraction step opens a PDF page with PyMuPDF, calls get_text(), and normalizes whitespace before indexing the resulting text.
