Friendly AI is an AI system intended not to harm humans. In the context of superintelligence, the central concern is that immense capability can produce very different outcomes depending on whether its goals are good or misguided. section 9.2.6
Keeping goal specification under human control and separating an agent’s reward from its goal are proposed safeguards; the latter is meant to avoid wireheading. A broader research program would support safe, constructive deployment. section 9.2.6
