liveBook · Manning

Precision

As explained in Self-Improving Agents

The accuracy with which a signal reports the size of the quality gap between an output and the desired result.

Precision measures how accurately a signal reports the gap between an agent’s output and desired quality. Repeated verdicts should agree, and reflect real quality rather than incidental traits such as length or tone. section 1.4.1 section 2.2.1

Precision trades off with availability. Gold sets and human evaluators can be precise but costly or slow to obtain; an LLM-as-Judge is widely available but noisier. Choose the most precise signal available for the task. section 2.2.1 section 1.4.1

Different signals, different precision

A labelled answer provides a highly precise signal. A unit test can deterministically pass or fail, while a model judge gives an opinion that may be inconsistent or biased. section 1.4.1 section 2.2.1

Figure 2.3 scatter plot demonstrating how precision and availability can guide your determination of how to evaluate good output from an agent

Compare evaluation signals by how precisely they measure output quality and how readily they can be obtained. section 2.2.1

How Self-Improving Agents covers it

How deep it goes
Defines precision through repeatable verdicts and alignment with real quality, then contrasts it with availability.
Examples
Uses labelled answers, gold sets, human evaluators, unit tests, and model judges as signal examples.
What you can do after
Evaluate output-quality signals by balancing measurement precision against their availability for the task.
How it connects
Connects precision with availability, ground truth, LLM-as-Judge, and process signals.

This explainer was generated by Manning's knowledge graph from the text of Self-Improving Agents. The section links point to the parts of the book it draws on.