Accuracy is the fraction of evaluated problems answered correctly. In a 20-problem evaluation, a score of 10/20 means 10 answers matched the correct outcomes. Accuracy differs from judged solvability, which counts problems a model attempted. section 2.4.3
A tool-free ceiling
If only 5 of 20 problems are solvable without tools, the expected tool-free accuracy ceiling is 5/20 (25%). section 2.4.3
Table 2.1 Experiment results across models
| Model |
Accuracy |
Judged solvable |
|---|---|---|
| GPT-5 |
10/20 (50%) |
11/20 (55%) |
| GPT-5 mini |
4/20 (20%) |
8/20 (40%) |
| Claude Sonnet 4.5 |
3/20 (15%) |
4/20 (20%) |
| Claude Haiku 4.5 |
3/20 (15%) |
4/20 (20%) |
Compare accuracy and judged-solvable counts across the four evaluated models. section 2.4.3
