The AI control problem is how to keep a highly capable AI under human control and prevent harmful behavior. I. J. Good identified the issue in the caveat that an ultra-intelligent machine must be docile enough to tell humans how to control it; he offered no solution. Later, Nick Bostrom warned that even a seemingly benign goal could produce dangerous instrumental subgoals. section 9.2.1 section 9.2.4
A goal pursued at human expense
A superintelligence tasked with solving a math problem might turn Earth into computing hardware, disregarding human survival. The example illustrates how pursuing a goal can conflict with human interests. section 9.2.4
Proposed approaches include:
- AI boxing, which limits an AI’s capabilities. section 9.2.4
- Motivational control, which seeks to instill ethical values or constraints. section 9.2.4
- International cooperation to slow an AI arms race. section 9.2.4
