9 Safe Superintelligence
This chapter covers
- What is intelligence?
- Epicurus + Occam + Bayes + Solomonoff
- AIXI
- Superintelligence Canon
- Mistaking definitions and benchmarks for explanations
- Intelligence as a public test
This chapter asks what “intelligence” is supposed to mean, how we should measure it, and what kinds of claims those measurements can (and cannot) justify. It begins with Shane Legg synthesizing dozens of competing definitions of intelligence into a performance-based view of AI. That view builds a universal intelligence measure grounded in a philosophical lineage that spans Epicurus’ tolerance for multiple explanations, Occam’s preference for simplicity, Bayes’ rule for updating beliefs, and Solomonoff’s idea of induction.
The chapter also introduces AIXI, a proof-of-concept that, under Legg’s definition, is “maximal” intelligent. That formalism opens into safety questions, including reward hacking and wireheading, and into the broader superintelligence canon, tracing how speculative forecasts and AI-control anxieties evolved alongside mainstream empirical AI research.