A RAG system can retrieve a passage containing the answer and still fail if that passage is buried among irrelevant results. Ranking addresses this “Missed the Top Rank” failure: after retrieval fetches candidates, a scoring model filters noise and prioritizes evidence for generation. Offline indexing makes documents searchable; ranking happens in the online query pipeline. section 1.2
In a semantic retrieval stack, encoding represents queries and documents, similarity search finds candidates, and ranking orders the results by relevance. These are distinct jobs: finding potentially useful documents does not establish which should receive the generator’s attention. section 2.4.1
Rank before compressing
Suppose a question asks whether a supplier’s indemnification obligations fall under a liability cap. A query-aware compressor scores four retrieved contract articles, drops an irrelevant notices clause, and moves the liability-cap article to the front. The resulting prompt uses 212 tokens rather than 347. But compression must preserve the cap’s exception for indemnification: a displayed alternative compressed passage loses that distinction and produces the wrong answer. Inspect retained evidence and the final answer, not just the token savings. section 13.2.4 section 13.2.3
Listing 13.1 LLMLingua.
from llmlingua import PromptCompressor
from typing import List, Dict, Any
def compress_with_longllmlingua(
query: str,
documents: List[str],
rate: float = 0.5
) -> Dict[str, Any]:
"""Compress documents with contrastive perplexity."""
compressor = PromptCompressor( #A
model_name="Qwen/Qwen2.5-0.5B",
use_llmlingua2=False,
device_map="cpu"
)
compressed_output = compressor.compress_prompt( #B
documents,
question=query,
rate=rate,
condition_compare=True,
condition_in_question="after",
reorder_context="sort",
dynamic_context_compression_ratio=0.4,
rank_method="longllmlingua"
)
return compressed_outputThe LLMLingua call shows query-conditioned compression with coarse ranking, document reordering, and dynamic token budgeting configured together. section 13.2.4
A separate reranker is not the only design. RankRAG instruction-tunes one model on both passage-relevance judgments and question answering, allowing the generator to rank its own context. Its training format supports a binary judgment for one passage or identification of relevant passages in a group. This can remove a separate ranking model, but requires ranking examples alongside QA data; MS MARCO passages are given as a general-purpose source. Choose the architecture by checking answer quality as well as which component and training data it requires. section 15.3.2 section 15.3








