liveBook · Manning

Goal specification

As explained in Sutskever's List

The process of formally defining the objectives an agent is intended to pursue.

Goal specification defines the objectives an agent is intended to pursue. In discussions of safe superintelligence, it is a safeguard: keep goals under human control and separate the agent from its reward and goal to reduce the risk of wireheading. section 9.2.6

The stakes rise with an agent’s capabilities. If its goals are good, superintelligence could bring major benefits; if they are bad or misguided, those same capabilities could contribute to catastrophe. Friendly AI and friendly AIXI-based frameworks are proposed as approaches intended to avoid harming humans. section 9.2.6

How Sutskever's List covers it

How deep it goes
Recommends human control and reward-goal separation to avoid wireheading, without detailing implementation.
What you can do after
Understand why human control and reward-goal separation are proposed as safeguards against harmful outcomes.
How it connects
Connects goal specification to reward design, wireheading, Friendly AI, and AIXI.
Cover of Sutskever's List
liveBookmanning.com
Read Sutskever's List on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.