Accuracy measures whether a model’s answers match known information or correct labels. In language-model training and evaluation, rewarding accuracy when answering while giving little or no reward for refusal can favor attempts over withholding uncertain answers. section 6.1.2
Measured accuracy can also depend on prompt details unrelated to the task’s meaning. Changes to formatting, wording, or example order have sometimes shifted classification predictions from correct to incorrect, or vice versa. section 6.3.2
Figure 6.1 Models learn patterns from training data and extend them to new cases. In this example, animals associated with water are often labeled as fish. When no clear example is available, as with the dolphin, the model does not stop or admit uncertainty. It continues by extending the strongest available pattern, incorrectly classifying the dolphin as a fish rather than a mammal.

A dolphin classified as a fish illustrates how extending a learned pattern can produce an incorrect label when a clear example is unavailable. section 6.1.2
















































