chapter seven

7 The Pivot to Reasoning

 

This chapter covers

  • VLAE: lossy latents with a PixelCNN decoder and an autoregressive prior
  • Relation Networks: pairwise relational reasoning on CLEVR and bAbI
  • Message Passing Neural Networks: graph learning and chemical accuracy on QM9
  • Relational Memory Core: attention among memory slots for sequential reasoning
  • Limits and Living doubts
Papers
  • Variational Lossy Autoencoder (Chen, 2016)
  • A Simple Neural Network Module for Relational Reasoning (Santoro, 2017)
  • Neural Message Passing for Quantum Chemistry (Gilmer, 2017)
  • Relational Recurrent Neural Networks (Santoro, 2018)

Scaling pretraining gave us more coherent text generation, but it did not provide dependable multi-step reasoning or planning. OpenAI’s trajectory after GPT-4 (2023) reflects this reality. Rather than scaling to GPT-5 with trillions of parameters, OpenAI invested in techniques like “test-time compute” that involve models “thinking” more at inference and fine-tuning with human feedback, essentially seeking algorithmic and architectural improvements beyond brute force.

7.1 Variational Lossy Autoencoder

7.1.1 What is VLAE?

7.1.2 Results

7.1.3 Autoregression Prior

7.1.4 Representation Learning

7.2 Relational Reasoning

7.2.1 Benchmark Evaluation

7.2.2 Impact

7.3 Neural Message Passing for Quantum Chemistry

7.3.1 Training and the QM9 Benchmark

7.3.2 Towers Architecture

7.3.3 Reception and Influence

7.4 Relational Recurrent Neural Networks

7.4.1 Experimental Methodology and Empirical Results

7.4.2 Historical Context

7.5 Modern Approaches to Reasoning

7.5.1 Killer Prompt

7.5.2 Reversal Curse

7.5.3 Scaffolding

7.6 Paper and Living Doubts