ARC-AGI is a benchmark dataset for testing whether an AI system can infer unfamiliar rules and apply them to new problems, rather than memorize known strategies. Each task provides several input-output grid examples, followed by a new input whose output must be constructed. Tasks are designed to measure generalization, abstraction, and on-the-fly reasoning. section 19.3.4
A task as a rule-inference problem
Suppose the examples show colored objects moving, mirroring, or being rearranged across grids. The solver must infer the transformation from the input-output pairs, apply that rule to a brand-new grid, and produce the correct output within three attempts. The challenge lies in discovering each task's unique logic, not recalling a previously practiced answer. section 19.3.4
Use ARC-AGI to distinguish several capabilities:
- Test generalization by withholding the evaluated tasks from training and programming. section 19.3.4
- Control the starting point by assuming shared Core Knowledge priors rather than treating every solver as knowledge-free. section 19.3.4
- Expose memorization limits: scaling base LLMs from GPT-2 to GPT-4.5 produced only roughly 0% to 10% performance on the 2019 benchmark version. section 19.3.5
- Reward flexible test-time methods, including parameter updates, search, chain-of-thought synthesis, and program synthesis. section 19.3.6
