Citation reliability measures whether an answer’s key claims are supported by the available evidence and whether each citation connects to the claim it supports. It makes evidence alignment a trackable evaluation criterion rather than a matter of impression. section 10.2.4
Evaluate citation reliability with a binary rubric:
- PASS when key claims are supported by sources and citations connect to those claims. section 10.2.4
- FAIL when sources are absent, citations differ from the original text, sources are unrelated, or an excerpt does not support the cited claim. section 10.2.4 section 10.3.2
- Judge only the supplied evidence pack; treat unsupported claims as failures even when they sound plausible, and give no partial credit. section 10.3.2
Evaluator input and decision
Provide the evaluator with the user query, final answer, rubric, and an evidence pack containing source identifiers, metadata, and relevant verbatim excerpts. If the answer cites a source whose excerpt does not support a key claim, return citation reliability = FAIL with a citation-mismatch reason. section 10.3.2
Aggregate case-level judgments into a citation reliability rate: the fraction of evaluations that PASS this rubric item. Combine it with answer relevancy and requirement compliance to calculate overall pass rate. section 10.2.4
