chapter seven

7 The Pivot to Reasoning

 

This chapter covers

  • Variational Lossy Autoencoders
  • Relation Networks on CLEVR and bAbI
  • Message Passing Neural Networks on QM9
  • Relational Memory Core for sequential reasoning across benchmarks
  • Paper versus Living Doubts
Papers
  • Variational Lossy Autoencoder (2016) Chen et al.
  • A Simple Neural Network Module for Relational Reasoning (2017) Santoro et al.
  • Neural Message Passing for Quantum Chemistry (2017) Gilmer et al.
  • Relational Recurrent Neural Networks (2018) Santoro et al.

Rather than scaling to GPT-5 with trillions of parameters, OpenAI invested in techniques such as “test-time compute,” which involves models “thinking” more during inference.[1] This pivot reflected a broader realization across the field that scaling pretraining alone was yielding diminishing returns, particularly on tasks requiring reasoning. In fact, even the loudest champions of scale began discussing its limits. In 2024, Ilya said that “the 2010s were the age of scaling; now we’re back in the age of wonder and discovery once again.” Sutskever added, “Scaling the right thing matters more now than ever.”[2]

7.1 Variational Lossy Autoencoder

7.1.1 What is VLAE?

7.1.2 Results

7.1.3 Autoregression Prior

7.2 Relational Reasoning

7.2.1 Benchmark Evaluation

7.3 Neural Message Passing for Quantum Chemistry

7.3.1 Accuracy as a Bottleneck

7.3.2 Message Passing Neural Networks

7.3.3 Training and Results

7.3.4 Towers architecture

7.3.5 Influence

7.4 Relational Recurrent Neural Networks

7.4.1 Experimental Methodology and Empirical Results

7.4.2 Historical context

7.5 Reasoning Models

7.5.1 Distributional Brittleness

7.5.2 Reversal Curse

7.5.3 Prompting as a Reasoning Interface

7.6 Paper Doubts