liveBook · Manning

Visual retrieval

As explained in Retrieval Augmented Generation, The Foundational Ideas

A visual retriever is a retrieval component that finds relevant documents or pages from visual representations rather than text alone.

Visual retrieval finds relevant document pages from rendered page representations instead of relying on extracted text alone. It addresses a production failure mode in which PDF text decodes to garbage and the information is lost before retrieval begins. section 15.6.5 section 15.9

Choose the architecture against the measured failure and operating budget:

  • Use page-image retrieval when visual layout, figures, or damaged extraction carries information that text retrieval cannot recover. section 15.6.5
  • Compare single-stage visual retrieval with text retrieval plus reranking rather than assuming the more sophisticated retriever wins every pipeline. section 15.6.5
  • Combine the top results from text and visual retrievers when complementary representations reduce retrieval mistakes, accepting additional context and storage costs. section 15.6.5 section 15.9
  • Budget late-interaction storage before scaling: roughly a thousand vectors per page can turn a few gigabytes into several terabytes for a million pages. section 15.6.5

Benchmark and production tradeoff

ViDoRe V3 reports visual retrieval at 59.8 nDCG@10 versus 51.0 for text in a single stage. Adding a reranker raises text to 63.6, while the visual reranker adds only 0.2 and degrades performance on four datasets. Hybrid retrieval reaches 65.7, exceeding the strongest text-only and visual-only systems, but gold-page evaluation shows that fusion can add cost when retrieval was already correct. section 15.6.5

Listing 15.5 Answering from the page image instead of the extracted text.
import base64
import pymupdf
from langchain_core.messages import HumanMessage
from providers import get_chat_model
 
def page_message(pdf, page_no, question, dpi=200, keep=1.0):
    page = pymupdf.open(pdf)[page_no]
    box = page.rect
    clip = pymupdf.Rect(box.x0, box.y0, box.x1,
                        box.y0 + box.height * keep)  #A
    pix = page.get_pixmap(dpi=dpi, clip=clip)  #B
    png = pix.tobytes("png")
    return HumanMessage(content=[
        {"type": "text", "text": question},
        {"type": "image", "source_type": "base64",
         "mime_type": "image/png",
         "data": base64.b64encode(png).decode()},  #C
    ])
 
QUESTION = ("Which tasks does the figure show, and what example "
            "input does it give for each?")
message = page_message(PAPER, 1, QUESTION, keep=0.42)  #D
print(get_chat_model(temperature=0).invoke([message]).content)

When indexing changes are unnecessary, render a selected PDF region at 200 dpi and send it as an image block alongside the question so the model can read content destroyed by extraction. section 15.9

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains architecture, benchmark tradeoffs, hybrid fusion, and storage arithmetic, but does not detail retriever training.
Examples
Uses ViDoRe V3 scores, million-page storage estimates, and a PyMuPDF page-image prompt.
What it assumes
Assumes a retrieval-augmented generation pipeline and teaches page rendering as a fallback when parsing fails.
What you can do after
Apply it via ColPali-family retrieval, hybrid search, or a PyMuPDF image fallback after diagnosing extraction failures.
How it connects
Connects visual retrieval with text retrieval, reranking, hybrid search, and late interaction.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.