liveBook · Manning

Confidence

As explained in Retrieval Augmented Generation, The Foundational Ideas

A confidence score is a numerical indication of a system's estimated certainty in an answer or prediction.

A confidence score is a scalar estimate of how strongly retrieved evidence supports an answer or prediction. In retrieval-augmented generation, a specialized evaluator scores document relevance before answer synthesis and uses the result to control the next operation. section 11.2.1

A retrieval system can turn the score into three operational zones:

  • High confidence: treat the retrieved documents as relevant and proceed to knowledge refinement. section 11.2.1
  • Low confidence: discard potentially misleading documents and trigger external web search. section 11.2.1
  • Intermediate confidence: retain the internal documents while using web search to corroborate or supplement them. section 11.2.1

Confidence is an audit signal, not proof that the best evidence was retrieved. The evaluator sees only the returned set, so a stale or incomplete corpus can still receive a plausible verdict. Marginalized evidence offers a complementary signal: agreement across documents raises confidence, while scattered support lowers it. section 11.2.1 section 4.2.2

Compliance retrieval

When a high-risk entity search returns ambiguous results, a confidence-aware system can avoid treating the internal passages as sufficient and use hybrid corroboration instead. section 11.2.1

Figure 11.3 CRAG Architecture: Using a retrieval evaluator to trigger corrective actions.

Use the architecture diagram to trace how evaluator output selects refinement, rejection with web search, or hybrid retrieval. section 11.2.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Descends from scalar scoring to three action branches, but stops short of guaranteeing unseen or higher-ranked evidence was retrieved.
Examples
Uses high-risk compliance retrieval, three confidence zones, and multi-document agreement as concrete illustrations.
What it assumes
Assumes retrieved documents and query-document relevance, while teaching confidence zones and their operational consequences.
What you can do after
Diagnose retrieval reliability by interpreting scores and routing low-confidence results toward refusal, web search, or corroboration.
How it connects
Connects retrieval evaluators, knowledge refinement, web search, marginalization, confidence calibration, and retrieval-augmented generation.

Also explained in

Compare how 16 books explain confidence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.