chapter nine
9 Safe Superintelligence
This chapter covers
- What is intelligence?
- Epicurus + Occam + Bayes + Solomonoff
- AIXI is introduced as a theoretical ceiling for Legg’s universal intelligence measure
- Superintelligence Canon (Good, Vinge, Kurzweil, Goertzel, Pennachin, Yudkowsky, and Bostrom)
- Mistaking definitions and benchmarks for explanations
- Wittgenstein and Turing reframe intelligence as public tests
In 2008, Shane Legg’s doctoral dissertation, Machine Super Intelligence, offered an explicit definition of intelligence, proposed a way to measure it, and asked what a “superintelligence” would be in that formal sense. Legg defined intelligence as “an agent’s ability to achieve goals in a wide range of environments” and argued that, in principle, this capacity can be summarized by a single scalar quantity. To construct his measure, Legg draws on Epicurus, Occam, Bayes, and Solomonoff. This yields a “universal prior” and, from it, a universal intelligence measure defined as the expected reward an ideal agent can accumulate across all computable environments.