liveBook · Manning

Accuracy

As explained in Machines that Think

Accuracy is the degree to which outputs correctly correspond to the intended facts, labels, or answers.

Accuracy measures whether a model’s answers match known information or correct labels. In language-model training and evaluation, rewarding accuracy when answering while giving little or no reward for refusal can favor attempts over withholding uncertain answers. section 6.1.2

Measured accuracy can also depend on prompt details unrelated to the task’s meaning. Changes to formatting, wording, or example order have sometimes shifted classification predictions from correct to incorrect, or vice versa. section 6.3.2

Figure 6.1 Models learn patterns from training data and extend them to new cases. In this example, animals associated with water are often labeled as fish. When no clear example is available, as with the dolphin, the model does not stop or admit uncertainty. It continues by extending the strongest available pattern, incorrectly classifying the dolphin as a fish rather than a mammal.

A dolphin classified as a fish illustrates how extending a learned pattern can produce an incorrect label when a clear example is unavailable. section 6.1.2

How Machines that Think covers it

How deep it goes
Explains reward asymmetry and prompt sensitivity, but does not provide a formula for calculating accuracy.
Examples
Uses a dolphin mislabeled as a fish and studies that vary prompt formatting, punctuation, and example order.
What you can do after
Evaluate model behavior by checking answers against known labels and comparing results across prompt variations.
How it connects
Connects accuracy to hallucination, refusal, fine-tuning, and prompt sensitivity.

Also explained in

Compare how 49 books explain accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.