Visual retrieval finds relevant document pages from rendered page representations instead of relying on extracted text alone. It addresses a production failure mode in which PDF text decodes to garbage and the information is lost before retrieval begins. section 15.6.5 section 15.9
Choose the architecture against the measured failure and operating budget:
- Use page-image retrieval when visual layout, figures, or damaged extraction carries information that text retrieval cannot recover. section 15.6.5
- Compare single-stage visual retrieval with text retrieval plus reranking rather than assuming the more sophisticated retriever wins every pipeline. section 15.6.5
- Combine the top results from text and visual retrievers when complementary representations reduce retrieval mistakes, accepting additional context and storage costs. section 15.6.5 section 15.9
- Budget late-interaction storage before scaling: roughly a thousand vectors per page can turn a few gigabytes into several terabytes for a million pages. section 15.6.5
Benchmark and production tradeoff
ViDoRe V3 reports visual retrieval at 59.8 nDCG@10 versus 51.0 for text in a single stage. Adding a reranker raises text to 63.6, while the visual reranker adds only 0.2 and degrades performance on four datasets. Hybrid retrieval reaches 65.7, exceeding the strongest text-only and visual-only systems, but gold-page evaluation shows that fusion can add cost when retrieval was already correct. section 15.6.5
Listing 15.5 Answering from the page image instead of the extracted text.
import base64
import pymupdf
from langchain_core.messages import HumanMessage
from providers import get_chat_model
def page_message(pdf, page_no, question, dpi=200, keep=1.0):
page = pymupdf.open(pdf)[page_no]
box = page.rect
clip = pymupdf.Rect(box.x0, box.y0, box.x1,
box.y0 + box.height * keep) #A
pix = page.get_pixmap(dpi=dpi, clip=clip) #B
png = pix.tobytes("png")
return HumanMessage(content=[
{"type": "text", "text": question},
{"type": "image", "source_type": "base64",
"mime_type": "image/png",
"data": base64.b64encode(png).decode()}, #C
])
QUESTION = ("Which tasks does the figure show, and what example "
"input does it give for each?")
message = page_message(PAPER, 1, QUESTION, keep=0.42) #D
print(get_chat_model(temperature=0).invoke([message]).content)When indexing changes are unnecessary, render a selected PDF region at 200 dpi and send it as an image block alongside the question so the model can read content destroyed by extraction. section 15.9
