Model calibration checks whether confidence signals or retrieval judgments agree with externally observed outcomes or labels. It makes those signals useful for monitoring and deciding when a retrieval system should correct its context. section 9.2.3 section 11.2.4
For reflection tokens, sample production traces and have a human or stronger model label the same passages and sentences. Compare IsRel with precision and recall, IsSup with agreement by class, and IsUse with rank correlation. A critic that marks unsupported sentences as fully supported is overconfident; raise the IsSup weight or retune on corrected examples. section 9.2.3
For retrieval, thresholds determine whether a system trusts its documents, checks further, or searches elsewhere. Tune sentence-level thresholds separately from document-level ones, but recognize when the scorer ranks relevant corrections incorrectly: no cutoff can recover what is ordered below unrelated material. An evaluator prompted to treat contradictions as relevant may address that failure. Miscalibration can silently certify stale information. section 11.2.4
An overconfident support token
If Self-RAG marks a sentence fully supported but human reviewers find it unsupported, the critic is overconfident. Compare the token against external labels, then raise its weight or retune on corrected examples. section 9.2.3
Listing 11.2 The CRAG pipeline.
from typing import List, Tuple
from enum import Enum
class RetrievalAction(Enum):
"""Defines the possible next steps based on retrieval quality."""
CORRECT = "correct"
AMBIGUOUS = "ambiguous"
INCORRECT = "incorrect"
class RetrievalEvaluator:
"""Audits retrieved documents and determines the pipeline action."""
def __init__(self, model, threshold_high: float = 0.85,
threshold_low: float = 0.45):
self.model = model
self.high_conf = threshold_high
self.low_conf = threshold_low
def evaluate(
self, query: str, documents: List[dict]
) -> Tuple[RetrievalAction, float]:
"""Scores documents and decides on the corrective action."""
if not documents:
return RetrievalAction.INCORRECT, 0.0
scores = [ #A
self.model.predict_score(query, doc['content']) #B
for doc in documents
]
agg_score = max(scores) if scores else 0.0 #C
if agg_score >= self.high_conf:
return RetrievalAction.CORRECT, agg_score #D
elif agg_score <= self.low_conf:
return RetrievalAction.INCORRECT, agg_score #E
return RetrievalAction.AMBIGUOUS, agg_score
def refine_knowledge(self, query: str, documents: List[dict]) -> str:
"""Filters irrelevant segments from retrieved documents."""
refined_context = []
for doc in documents:
strips = doc['content'].split(". ") #F
for strip in strips:
score = self.model.predict_score(query, strip)
if score > 0.5: #G
refined_context.append(strip)
return "\n".join(refined_context)
Inspect the retrieval evaluator's thresholds and how its confidence score selects a corrective action. section 11.2.4






