Model calibration is the agreement between a model’s solvability judgments and what it can actually answer. In a 20-problem evaluation, only five problems were solvable without tools. Models judging few problems solvable were often closer to that count; more ambitious models sometimes answered questions using information encountered during training, but could not reliably distinguish accurate knowledge from confabulation. section 2.4.3
Table 2.1 Experiment results across models
| Model |
Accuracy |
Judged solvable |
|---|---|---|
| GPT-5 |
10/20 (50%) |
11/20 (55%) |
| GPT-5 mini |
4/20 (20%) |
8/20 (40%) |
| Claude Sonnet 4.5 |
3/20 (15%) |
4/20 (20%) |
| Claude Haiku 4.5 |
3/20 (15%) |
4/20 (20%) |
Compare each model’s accuracy with the number of problems it judged solvable. section 2.4.3
