Precision measures how accurately a signal reports the gap between an agent’s output and desired quality. Repeated verdicts should agree, and reflect real quality rather than incidental traits such as length or tone. section 1.4.1 section 2.2.1
Precision trades off with availability. Gold sets and human evaluators can be precise but costly or slow to obtain; an LLM-as-Judge is widely available but noisier. Choose the most precise signal available for the task. section 2.2.1 section 1.4.1
Different signals, different precision
A labelled answer provides a highly precise signal. A unit test can deterministically pass or fail, while a model judge gives an opinion that may be inconsistent or biased. section 1.4.1 section 2.2.1
Figure 2.3 scatter plot demonstrating how precision and availability can guide your determination of how to evaluate good output from an agent

Compare evaluation signals by how precisely they measure output quality and how readily they can be obtained. section 2.2.1
