liveBook · Manning

Retrieval evaluator

As explained in Retrieval Augmented Generation, The Foundational Ideas

A retrieval evaluator is a component that assesses retrieval results and can trigger searches for additional supporting sources.

A retrieved passage can look relevant and still support the wrong answer. A compliance search, for example, might return a 2020 policy on expedited due diligence while missing a 2024 directive that overrides it. A retrieval evaluator puts a quality gate between retrieval and answer generation, so doubtful evidence can trigger another search rather than flow straight into an answer. section 11.1.3 section 11.2.4

In corrective retrieval-augmented generation (CRAG), the evaluator is separate from the answer-generating model. It scores how relevant a retrieved document is to a query and uses confidence thresholds to select the next action. It may be implemented with a fine-tuned T5 model, a compact model, a cross-encoder, or a calibrated LLM judge. section 11.2.1 section 11.2.4

The score routes retrieval into three paths:

  • High confidence: retain the internal evidence and refine it for the generator. section 11.2.1
  • Low confidence: discard misleading results and search externally for better evidence. section 11.2.1
  • Intermediate confidence: keep internal evidence but search externally to corroborate or supplement it. section 11.2.1

An ambiguous compliance result

For a question about expedited onboarding of an EU politically exposed person, an internal 2020 policy scores 0.76, a 2024 directive scores 0.63, and an unrelated holiday policy scores 0.00. The result is classified as ambiguous, so the pipeline uses a hybrid source path instead of trusting the top internal hit. Thresholds such as 0.85 and 0.45 are starting points; tune them on held-out questions because changing them changes which path runs. section 11.2.4

The evaluator can grade only documents it receives. High confidence does not prove that a better, newer document was not ranked below the retrieval cutoff. Publication dates, supersession links, and retiring obsolete documents address that upstream problem; external search offers a route beyond the internal index when the evaluator flags doubt. Web-search fallback also adds latency, so its value depends on the cost of a wrong answer. section 11.2.1 section 9.5.1 section 11.2.5

Listing 11.3 The CRAG orchestrator.
class CRAGPipeline:
    """Orchestrates the retrieve-evaluate-correct loop."""
 
    def __init__(self, retriever, evaluator, web_searcher):
        self.retriever = retriever
        self.evaluator = evaluator
        self.web_searcher = web_searcher
 
    def run(self, query: str) -> dict:
        """Executes the corrective RAG workflow."""
        internal_docs = self.retriever.retrieve(query)  #A
 
        action, score = self.evaluator.evaluate(query, internal_docs)  #B
 
        context = ""
        source = "internal"
 
        if action == RetrievalAction.INCORRECT:  #C
            web_docs = self.web_searcher.search(query)  #D
            context = self.evaluator.refine_knowledge(query, web_docs)
            source = "web"
 
        elif action == RetrievalAction.AMBIGUOUS:
            internal_ctx = self.evaluator.refine_knowledge(  #E
                query, internal_docs)
            web_docs = self.web_searcher.search(query)
            web_ctx = self.evaluator.refine_knowledge(query, web_docs)
 
            context = f"{internal_ctx}\n\n{web_ctx}"  #F
            source = "hybrid"
 
        else:
            context = self.evaluator.refine_knowledge(query, internal_docs)  #G
 
        return {"context": context, "action": action.value,
                "source": source, "score": score}
 
 

The Python orchestrator shows where evaluation sits after retrieval and how an incorrect verdict triggers external search. section 11.2.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines relevance scoring, three confidence zones, pipeline branching, and threshold tuning; identifies unseen retrieved-out documents as a limit.
Examples
Uses conflicting 2020 and 2024 compliance sources, scored retrieval results, and a Python retrieve-evaluate-correct pipeline.
What it assumes
Assumes familiarity with a RAG pipeline’s retriever and answer generator while defining the evaluator’s distinct role.
What you can do after
Build it from scratch: separate scoring from orchestration, tune confidence thresholds, and route uncertain retrieval to external search.
How it connects
Connects corrective retrieval-augmented generation to knowledge refinement, web search, Self-RAG, and document supersession.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.