A retriever can find passages that look relevant without putting the passage that answers the question first. Reranking addresses that gap: retrieve a manageable candidate set quickly, score those candidates more carefully against the query, and pass the highest-ranked passages to the answer generator. section 8.6 section 15.7.2
The usual first stage is a bi-encoder: it computes document vectors ahead of time and uses a nearest-neighbor lookup at query time. A cross-encoder instead reads the query and one candidate passage together and returns a relevance score. That joint reading can improve precision, but scoring an entire corpus this way would require a model pass for every document. Run it only on the first stage’s top few dozen candidates. section 8.6 section 7.6.3
The candidate set can come from several retrieval designs:
- HyDE can retrieve a broad set, such as 50 documents, before a cross-encoder selects the strongest matches. section 7.6.3
- RAG-Fusion can merge results from multiple query variations before reranking the survivors; broader coverage adds retrieval work and latency. section 1.5.2 section 8.6
- An updated retriever can rerank a larger set fetched from a stale index by re-embedding only those candidates, avoiding a full index rebuild at query time. section 6.4.1
Table 6.5 Update strategies
|
Strategy
|
Mechanism
|
Computational Cost
|
Use Case
|
Key Trade-off
|
|
Full Index Update
|
Re-embed the entire corpus periodically.
|
Very High
|
Full-dataset fine-tuning; when accuracy is paramount.
|
Highest potential accuracy at the cost of
downtime/overhead.
|
|
Re-ranking
|
Retrieve
N≫k with stale index;
re-rank N live.
|
Low
|
Balancing freshness and efficiency; production systems.
|
Assumes the best documents are in the initial N
candidates.
|
|
Query-Side Fine-tuning
|
Freeze document encoder; only train query encoder.
|
Very Low
|
Few-shot learning (e.g., <1000 examples); rapid
adaptation.
|
Potentially lower peak accuracy with large datasets.
|
Compare live reranking against full index updates and query-side fine-tuning. Its key constraint is visible in the table: a reranker cannot recover a relevant document missing from the initial candidates. section 6.4.1
A precise match that misses the answer
In a fintech question-answering system, cross-encoder reranking helped when initial results were semantically similar but factually distinct. It sometimes hurt narrow technical questions by promoting dense passages above plainer passages containing the answer. Check answer quality as well as passage relevance, and measure the added latency: a five-second research workflow may be acceptable internally but not for sub-second customer support. section 15.7.4
A separate reranker is not the only option. RankRAG trains a generator on both question-answering and passage-ranking examples so it can perform its own relevance assessment. Whether ranking uses that design or a cross-encoder, compare it with a fast baseline rather than assuming another stage always improves the result. section 15.3.2 section 15.7.4








