Abstention is an evaluation behavior that lets a judge decline to score when the available context cannot support a meaningful verdict. It prevents missing evidence from becoming a confident-looking but ungrounded judgment. Unlike escalation, which asks an uncertain judge to try harder, abstention recognizes that more reasoning or model strength cannot compensate for absent information. section 10.5 section 10.5.3
Implement abstention as an explicit path in the judging contract and the surrounding evaluation pipeline:
- Require the judge to score only when the supplied context is sufficient to verify the response. section 10.5.3
- Return a distinct status such as scored or abstained, with a null score and a reason identifying what information is missing. section 10.5.3
- Route abstained cases to a stronger judge with better context or to human review, while reporting abstention separately from escalation. section 10.5.3
- Monitor the defer rate as a possible data-pipeline signal, especially when it rises over time. section 10.7
Fact-checking with incomplete source context
A judge receives a query, response, and source context. If the context verifies every relevant claim, it returns {"status":"scored","score":1-5,"reason":"..."}. If a claim cannot be checked, it returns {"status":"abstained","score":null,"reason":"missing information needed to verify the claim"} and routes the case onward. section 10.5.3
