How deeply does Sutskever's List cover ARC-AGI-2?

Defines benchmark difficulty through novel tasks and evaluation reversals, but does not describe task mechanics or solver internals.

ARC-AGI-2

As explained in Sutskever's List

ARC-AGI-2 is a harder successor benchmark of grid-based abstraction and reasoning tasks designed to reduce exploitation of patterns from earlier ARC evaluations.

ARC-AGI-2 is a harder successor benchmark for grid-based abstraction and reasoning. It introduces novel tasks designed to defeat strategies specialized to ARC-AGI-1, testing whether systems infer rules rather than exploit benchmark-specific regularities. section 7.6

A sharp reversal

GPT-3 scored 0% on the public evaluation, while GPT-4.1 reached roughly 5%. OpenAI’s o3-preview later scored 75.7% on a semi-private evaluation under a $10,000 compute limit, but the same model family fell below 3% when ARC-AGI-2 appeared. The reversal exposed overspecialization rather than a solved problem of general abstraction. section 7.6

The benchmark operationalizes skepticism: increasing difficulty and task novelty distinguish genuine generality from learning quirks of a test. Its unresolved results make it a practical challenge for reasoning research. section 7.6

How Sutskever's List covers it

liveBook manning.com

Read Sutskever's List on liveBook Get the book at manning.com

Related concepts

Abstraction & Reasoning Corpus for Artificial General Intelligencecontrasts with · The Abstraction and Reasoning Corpus for Artificial General Intelligence is a benchmark of visual reasoning tasks designed to assess abstract rule induction and generalization from examples.
compositional generalizationmeasures · Compositional generalization is the ability to apply learned components or capabilities to novel combinations and tasks.
brute-force searchcontrasts with · Brute-force search systematically tests many or all candidates in a defined search space, without relying on selective shortcuts to narrow it.
fluid intelligencemeasures · Fluid intelligence is the ability to solve novel problems and adapt reasoning to unfamiliar situations without relying solely on previously learned knowledge.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.