liveBook · Manning

Reranking

As explained in Retrieval Augmented Generation, The Foundational Ideas

A reranker is a model or retrieval component that reorders candidate results according to estimated relevance.

A retriever can find passages that look relevant without putting the passage that answers the question first. Reranking addresses that gap: retrieve a manageable candidate set quickly, score those candidates more carefully against the query, and pass the highest-ranked passages to the answer generator. section 8.6 section 15.7.2

The usual first stage is a bi-encoder: it computes document vectors ahead of time and uses a nearest-neighbor lookup at query time. A cross-encoder instead reads the query and one candidate passage together and returns a relevance score. That joint reading can improve precision, but scoring an entire corpus this way would require a model pass for every document. Run it only on the first stage’s top few dozen candidates. section 8.6 section 7.6.3

The candidate set can come from several retrieval designs:

  • HyDE can retrieve a broad set, such as 50 documents, before a cross-encoder selects the strongest matches. section 7.6.3
  • RAG-Fusion can merge results from multiple query variations before reranking the survivors; broader coverage adds retrieval work and latency. section 1.5.2 section 8.6
  • An updated retriever can rerank a larger set fetched from a stale index by re-embedding only those candidates, avoiding a full index rebuild at query time. section 6.4.1
Table 6.5 Update strategies
Strategy
Mechanism
Computational Cost
Use Case
Key Trade-off
Full Index Update
Re-embed the entire corpus periodically.
Very High
Full-dataset fine-tuning; when accuracy is paramount.
Highest potential accuracy at the cost of downtime/overhead.
Re-ranking
Retrieve N≫k with stale index; re-rank N live.
Low
Balancing freshness and efficiency; production systems.
Assumes the best documents are in the initial N candidates.
Query-Side Fine-tuning
Freeze document encoder; only train query encoder.
Very Low
Few-shot learning (e.g., <1000 examples); rapid adaptation.
Potentially lower peak accuracy with large datasets.

Compare live reranking against full index updates and query-side fine-tuning. Its key constraint is visible in the table: a reranker cannot recover a relevant document missing from the initial candidates. section 6.4.1

A precise match that misses the answer

In a fintech question-answering system, cross-encoder reranking helped when initial results were semantically similar but factually distinct. It sometimes hurt narrow technical questions by promoting dense passages above plainer passages containing the answer. Check answer quality as well as passage relevance, and measure the added latency: a five-second research workflow may be acceptable internally but not for sub-second customer support. section 15.7.4

A separate reranker is not the only option. RankRAG trains a generator on both question-answering and passage-ranking examples so it can perform its own relevance assessment. Whether ranking uses that design or a cross-encoder, compare it with a fast baseline rather than assuming another stage always improves the result. section 15.3.2 section 15.7.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains joint query-passage scoring and candidate-set costs, then contrasts cross-encoder reranking with live rescoring from a stale index.
Examples
Uses a fintech retrieval evaluation, a 50-candidate HyDE pipeline, and a stale-index top-100 update strategy.
What it assumes
Assumes a retrieval pipeline while explaining the scoring difference between separately encoded vectors and jointly read query-passage pairs.
What you can do after
Apply it via Sentence Transformers’ retrieve-and-rerank notebook, comparing bi-encoder retrieval, cross-encoder ordering, and BM25.
How it connects
Links reranking to bi-encoders, cross-encoders, HyDE, RAG-Fusion, index updates, and retrieval-augmented generation.

Also explained in

Compare how 9 books explain reranking →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.