A fluent answer or a successful type check can still be wrong. Accuracy measures how often output is correct against a standard external to the model. For classification, it is the proportion of predictions that match the expected labels. Neither a model’s confidence nor structurally valid code proves correctness. section 1.3 section 8.2.1 section 4.6.3
Overall accuracy can hide the errors that matter most. A medicine-risk classifier reports over 94% accuracy, but precision and recall reveal different risks: false alarms and missed high-risk cases. Check those measures and the confusion matrix before treating the overall score as evidence of safety. section 4.6.3
Figure 4.10 The classification accuracy: confusion matrix shows that most (931) of the actual false values were predicted as false, and most of the true values (491) were predicted as true.

The confusion matrix separates correct safe and high-risk predictions, making the aggregate accuracy easier to interpret. section 4.6.3
Context is not a free accuracy upgrade
In a text-to-SQL benchmark, adding recent queries improved overlap similarity, but gains diminished. For moderate queries, the median rose from about 0.39 with no history to 0.56 with ten queries and 0.61 with twenty. Overlap similarity is the reported proxy here, not proof that generated SQL is correct. For this benchmark, ten queries offered the preferred accuracy-to-cost trade-off; twenty cost more for a smaller gain. section 6.4.1 section 6.4.2
More context can also make an answer less reliable. In one code-comprehension benchmark, a model scored 29% at 32,000 tokens and 3% at 256,000, even though the answer was present in both contexts. An advertised context window is therefore not a guarantee that all supplied information will be used reliably. section 3.4.1
Use accuracy as one part of a verification loop:
- Define an external standard and test representative cases; do not rely on fluent output or compilation alone. section 1.3 section 8.2.1
- Inspect error types and task-specific measures alongside the aggregate score, especially when false negatives and false positives have different costs. section 4.6.3
- Benchmark context choices against both quality and cost. Treat the best setting as specific to the workload rather than a universal maximum. section 6.4.1 section 6.4.2
















































